Spatio-Temporal Encoding of Brain Dynamics with Surface Masked Autoencoders

Authors: simon  dahan
logan z j williams
yourong  guo
daniel  rueckert
emma c robinson
l z j dahan
y  williams
d  guo
e c rueckert
  robinson
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 19:06:12

Abstract: The development of robust and generalisable models for encoding the spatio-temporal dynamics of human brain activity is crucial for advancing neuroscientific discoveries. However, significant individual variation in the organisation of the human cerebral cortex makes it difficult to identify population-level trends in these signals. Recently, Surface Vision Transformers (SiTs) have emerged as a promising approach for modelling cortical signals, yet they face some limitations in low-data scenarios due to the lack of inductive biases in their architecture. To address these challenges, this paper proposes the surface Masked AutoEncoder (sMAE) and video surface Masked AutoEncoder (vsMAE) - for multivariate and spatio-temporal pre-training of cortical signals over regular icosahedral grids. These models are trained to reconstruct cortical feature maps from masked versions of the input by learning strong latent representations of cortical structure and function. Such representations translate into better modelling of individual phenotypes and enhanced performance in downstream tasks. The proposed approach was evaluated on cortical phenotype regression using data from the young adult Human Connectome Project (HCP) and developing HCP (dHCP). Results show that (v)sMAE pre-trained models improve phenotyping prediction performance on multiple tasks by $\ge 26\%$, and offer faster convergence relative to models trained from scratch.  Finally, we show that pre-training vision transformers on large datasets, such as the UK Biobank (UKB), supports transfer learning to low-data regimes. Our code and pre-trained models are publicly available at https://github.com/metrics-lab/surface-masked-autoencoders.

Main paper: 
 introduction: 
 The construction of robust and generalisable AI models of human brain function remains a formidable challenge due to the high-dimensional, temporal fluctuations of human brain activations (Vidaurre et al., 2017; Pervaiz et al., 2022) . These patterns exhibit considerable heterogeneity across individuals, making it difficult to learn latent representations that generalise across individuals. Focusing specifically on the cerebral cortex, research has long shown that different areas perform different functions (Glasser et al., 2016) , and that highorder cognition arises from dynamic interactions between these regions (Owen et al., 2021) . For these reasons, several studies have chosen to model the brain as a graph (Bullmore and Sporns, 2009) and study functional dynamics using graph neural networks or sequence models (Dahan et al., 2021; Choi et al., 2023; Kim et al., 2023) . One limitation with this approach resides in the difficulty of delineating cortical functional areas from MRI. Most studies assign regions from a population-average atlas (Kim et al., 2021) ; however, this inserts noise and errors into the estimation of regional timeseries, since human brains cannot be perfectly spatially normalised to a template through diffeomorphic registration (Glasser et al., 2016) . Other studies instead prefer to treat brain activity independently at each voxel (Huth et al., 2016) , but this ignores the spatial coherence of signals, both from adjacent voxels that belong to the same region, as well as distantly connected areas whose time series are correlated with that region. Pioneering work by Fischl et al. (1999) , but later advanced by the HCP (Glasser et al., 2013 (Glasser et al., , 2016;; Coalson et al., 2018) , proved that analysis of cortical fMRI is most precise when treated as functions on a surface mesh. This suggests that encoding and decoding from fMRI might be improved by explicitly accounting for long-range spatial-temporal interactions across the cortical surface.Vision transformers (Dosovitskiy et al., 2020 ) (ViTs) have been established as a powerful tool for studying long-range dependencies in natural images, leveraging the mechanisms of self-attention to outperform CNNs across a range of image understanding tasks (Zong et al., 2023; Liu et al., 2022) . Unfortunately, such performance gains usually come at the cost of requiring very large data sets, to compensate for the relative lack of constraints on transformer architectures. To overcome this limitation, self-supervision frameworks have been developed, that pre-train networks on simpler tasks (Bao et al., 2022; Caron et al., 2021) . One recent popular approach has been the development of auto-encoder frameworks which seek to reconstruct whole images from inputs that have had the majority of the image corrupted (Dosovitskiy et al., 2020) or had patches masked out (He et al., 2021) .Recently we translated the concept of ViTs to the cortical surface, by proposing a surface patching scheme derived from tessellations of regular icosahedrons (Dahan et al., 2022) . Treating cortical modelling as a sequence-to-sequence learning problem was shown to outperform surface convolutional approaches on a range of phenotype prediction tasks (Zhao et al., 2019; Monti et al., 2016) . In this paper, we integrate the concept of MAE selfsupervision (He et al., 2021) into this Surface Vision Transformer (SiT) framework. Results show that the resulting (v)sMAEs learn robust and generalisable representations that significantly enhance phenotype prediction, support transfer-learning from large-open datasets (to support learning in low data regimes) and most importantly support reconstruction of cortical functional dynamics (with up to 75% missing data). 
related works: 
 This work extends from the Masked Autoencoder (MAE) (He et al., 2021) , which learns strong visual representations through modelling reconstruction of whole images from inputs which have had the majority of their features masked out. For ViTs, images are represented from a sequence of image patches (or tokens). The success of the MAE comes from its asymmetric encoder-decoder architecture, in which the encoder processes only a fraction (ρ) of the input sequence -the unmasked tokens -while the decoder learns to reconstruct the image at full resolution based solely on the embeddings learnt from these unmasked tokens. Such self-supervision facilitates the learning of robust and generalisable representations, since the complexity of the self-attention operation scales quadratically with the length of the input sequence -thus, passing fewer tokens allows for the building of much deeper encoder networks. As the objective is to use only the encoder for downstream tasks a light-weight decoder is considered sufficient for reconstruction, ensuring that the full framework remains computationally efficient. Extending this concept, VideoMAE models (Tong et al., 2022; Feichtenhofer et al., 2022) have sought to address the unique challenges of reconstructing spatio-temporal patches from successive video frames. The MAE framework has also been applied to non-Euclidean domains such as point clouds, irregular meshes and graphs (Hou et al., 2022; Liang et al., 2022) . The MAE approach to self-supervision contrasts with the masked patch prediction (MPP) model proposed by (Dosovitskiy et al., 2020) , which instead employs a symmetric encoder-decoder architecture that is trained to reconstruct the entire image sequence after corrupting some of the input patches through masking or swapping. This approach was adapted to the cortical surface domain in (Dahan et al., 2022) . However, since its inception the MPP has been shown to be repeatedly outperformed by the MAE, which demonstrates better reconstruction, efficiency, and performance on fine-tuning tasks. Unmasking strategies. 
surface vision transformer: 
 Following Dahan et al. (2022) , input cortical feature maps X ∈ R |V 6 |×C (C channels) are represented on a 6th-order icospheric (ico6) tessellation: 2) . This partition leads to a sequence of non-overlapping triangular patches:I 6 = (V 6 , F 6 ),T 3 = {t 1 3 , t 2 3 , ..t |F 3 | 3 } (with t i 3 ⊂ V 6 , |t i 3 | = 45 vertices in each patch).Imaging features (e.g. myelin and curvature) from ico6 vertices that fall within each patch are then concatenated across channels, flattened and projected with a trainable linear layer into a set of D-dimensional input tokens{X (0) i } N i=1 . Sine-cosine positional embeddings, E pos = {E i } Ni=1 are then added to each of the tokens to encode patch location within the sequence:EQUATIONN + E N , where each E i reflects a D-dimensional vector that encodes location from a unique combination of sine and cosine functions (Vaswani et al., 2017) (more details in Appendix A.4). Use of fixed positional embeddings, instead of the trainable embeddings used in (Dahan et al., 2022) , was found to speed up network training. The initial sequence X (0) is then processed by L consecutive transformer encoder blocks of Multi-Head Self-Attention (MHSA) and Feed Forward Network (FFN) layers, with residual layers in-between:EQUATIONNote, that sequence shape is preserved through each block. This Surface Vision Transformer (SiT) architecture forms the backbone for all (v)sMAE encoders and decoders. More information on surface patching and architecture details can be found in Appendix A.2.Implementation of the proposed sMAE parallels that of the original MAE architecture. First, unmasked tokens are randomly selected from the set of all possible patches available from an ico3 mesh ( (b) vsMAE reconstruction (ρ = 50%) results with 3 7T HCP consecutive frames.In the domain of video understanding, a range of spatiotemporal masking strategies have been explored for self-supervision with videoMAE. These include spacetime agnostic masking strategies (Feichtenhofer et al., 2022) , which randomly sample patches in a different way for each and every frame; as well as tube-masking strategies (Tong et al., 2022; Wang et al., 2023) , which sample concurrent frames from the same mask. When translating such concepts to the modelling of cortical functional dynamics it is important to acknowledge that, relative to natural video, fMRI is much less structured, with far less temporal redundancy. This is because fMRI is an indirect measure of cortical activity, characterised by low temporal resolution and corruption from physiological noise. Thus, while agnostic masking with high masking ratios (of up to 90%) (Feichtenhofer et al., 2022) may work well for natural scenes, it cannot be easily made to work for fMRI due to the noise. Nor can classic videoMAE implementations of tube-masking, since these compress video cube patches of size T × 16 × 16 with a single linear projection. This makes sense for natural videos, for which successive frames from the same patch will probably contain highly correlated features, but not for fMRI where two frames may present very distinct patterns of activity (due to the low temporal resolution, see Figure 2 .b). We address these challenges in two ways: first, we account for noise through the use of a tube-masking strategy with a reduced masking ratio (typically 50%, see Fig 2b) ; second, we change the approach for spatio-temporal patch compression from one projection across block of frames, to one projection per frame, followed by concatenation across frames. This design creates a spatiotemporal sequence of tokens and enables the vsMAE model to effectively compute spacetime self-attention, enhancing the model's capacity to capture and integrate complex spatio-temporal dynamics between patches distant in both space and time. 
experimental methods: 
 To evaluate whether the proposed (v)sMAE can learn sufficiently rich encodings of cortical functional dynamics a series of experiments was conducted: first, we validated whether the model could robustly generalise to unseen subjects by assessing the quality of the generated reconstructions; then, we investigated whether the encodings learned by our (v)sMAE models capture information relevant to phenotypic predictions and therefore may suggest an alignment with neurobiological patterns. In parallel, we demonstrate that self-supervised pre-training of (v)sMAEs on very large open-datasets, such as UK Biobank, can support transfer-learning to data sets of much more limited size.dHCP Data consists of cortical surface meshes and metrics (sulcal depth, curvature, cortical thickness and T1w/T2w myelination) derived from T1-and T2-weighted magnetic resonance images (MRI) from the developing Human Connectome Project (dHCP) (Makropoulos et al., 2018) . We use 580 scans from 419 term neonates (born after 37 weeks gestation) and 111 preterm neonates (born prior to 37 weeks gestation). Preterm babies were scanned either shortly after birth, at term equivalent age (TEA), or at both timepoints.UKB Matched cortical metrics were derived from 4063 subjects (1896 biological females) aged between 46 and 83 years in the UKB Biobank (UKB) dataset (Miller et al., 2016; Alfaro-Almagro et al., 2018) . These were used for the transfer learning experiment.HCP fMRI data was obtained from the movie-watching experiment of the HCP 7T release (Van Essen et al., 2013) , from 174 participants who were scanned while watching a series of movie clips. fMRI responses were projected from the volume to the cortical surface (Glasser et al., 2013) and were aligned using multimodal cortical features (MSMAll) (Robinson et al., 2014a (Robinson et al., , 2018a;; Glasser et al., 2016) . For these subjects, we also used Z-statistic contrast maps derived from the N-back Working Memory (WM) task (Braver et al., 1997; Barch et al., 2013a) from the 3T HCP release (Van Essen et al., 2013) . More details about data acquisition, processing and train/validation/test splits for all the datasets in Appendix A.1.(v)sMAE pretraining All models were trained on a single RTX 3090 NVIDIA GPU with Adam optimisation (LR = 3e -4 and cosine decay). A batch size of 16 was used by default but adapted depending on ρ values. The impact of different masking ratios (ρ) was evaluated by testing the quality of reconstructions (on all cortical features for sMAE and on 7T movietask data for vsMAE) and fine-tuning on phenotype regression tasks. The performance of the vsMAE was similarly optimised over frame sampling rates τ ∈ {1, 3, 6, 8}. Phenotyping Predictions Self-supervision with (v)sMAEs was validated by fine-tuning the pre-trained weights from the SiT encoders on various phenotyping prediction tasks. sMAE encoders, trained on multivariate dHCP cortical imaging features, were validated for regression of post-menstrual age (PMA) at scan, and gestational age (GA) at birth. The GA task aims to predict the degree of prematurity from scans acquired around TEA, while the prediction of PMA was designed as a correlate for modelling healthy cortical maturation.On fMRI data, we evaluated the performance of the vsMAE model on sex classification. Finetuning was found to perform best when using SGD optimisation (momentum=0.9, warmup scheduler) with LR = 1e -4 for sex classification, GA and PMA.Transfer learning We evaluate the potential use of (v)sMAE encoders for transfer learning by first training sMAEs on multivariate (sulc, curvature, cortical thickness and myelin) from UKB; the resulting encoder was then fine-tuned for dHCP PMA prediction. The vsMAE, pretrained on fMRI data, was fine-tuned on fluid intelligence prediction using contrast maps extracted from the HCP 3T working memory task. We used SGD optimisation (momentum=0.9, warmup scheduler) with LR = 1e -5 for fluid intelligence prediction. 
results & discussion: 
 Evaluating reconstruction of brain dynamics Results for the sMAE showed that masking ratios of ρ = 50% and 75% yielded the strongest visual reconstruction, lowest reconstruction error and highest performance on GA downstream tasks ( ). This suggests that modelling spatio-temporal dynamics improves cortical phenotype prediction.Transfer Learning Finally, to assess the potential of (v)sMAEs for transfer learning on smaller datasets, we investigated fine-tuning, using subsets of the entire dHCP dataset (10%, 20% or 50%) following training on all 4063 UKB datasets. Results in Figure 3 .b.show that, relative to training from scratch, transfer-learning improves cortical phenotype prediction on dHCP for all data ratios (orange line). Finetuning the vsMAE encoder on 3T HCP contrast maps similarly yielded higher correlation scores on the challenging fluid intelligence prediction task (0.39) Pearson correlation compared to training from scratch (≤ 0.3), see Appendix B.3.1.In this paper, we demonstrated that pre-training surface vision transformers with (v)sMAEs is an effective way to learn strong representations of both static and dynamic cortical maps. Training SiTs in this way leads to better performance on downstream phenotype prediction tasks irrespective of whether the self-supervision task is trained on the same data set, or larger-open data sets such as UKB (even when the data set demographics strongly diverge). This offers significant potential for translating the benefits of SiTs to much smaller clinical neuroimaging datasets (e.g. for psychosis (Demro et al., 2021) ). Moreover, the strong performance of vsMAE on reconstruction and phenotype prediction from fMRI suggests these models can learn robust and generalisable models of dynamic cortical function. This opens the door in future to novel applications in fMRI encoding and decoding.sMAE reconstruction task, a batch size of 128 is typically used, with 4 cortical input metrics. This allows for a fast training of the (v)sMAE models. In the present study, we patched the cortical surface using an ico3 tessellation grid. It achieved good phenotyping performance and allow for higher-resolution patch representation, compared to ico2 sampling as in (Dahan et al., 2022) , while preserving a manageable computational cost (batch size of 128 vs 256 for ico2). With the ico3 patching, the cortical surface is represented by 1280 patches of 45 vertices each, compared to 320 patches of 153 vertices for ico2.The sMAE methodology is compared to the Masked Patch Prediction (MPP) self-supervision task, used previously in (Dahan et al., 2022) , which in turn was adapted from (Dosovitskiy et al., 2020; Devlin et al., 2019) . It employs an autoencoder architecture that is trained to reconstruct the entire image sequence while corrupting some of the input patches through masking or swapping. Following (Dosovitskiy et al., 2020) , we corrupt 50% of the input patches randomly: by replacing the patches with a masked (empty) token (40%), using another patch embedding from the sequence at random (5%), or preserving their original embeddings (5%). In contrast to sMAE, the MPP methodology optimises reconstruction by computing the MSE loss for all patches, and the MPP encoder processes the entire sequence of patches, which reduces its efficiency and modelling power with long input sequences.Compared to (Dahan et al., 2022) , we use fixed positional embeddings, which accelerate the training process compared to learned positional embeddings. Positional embeddings are defined as follows:E i = P E (i,j) D j=1(2) where: EQUATIONWe evaluate the importance of positional embeddings, by comparing reconstructions with and without the use of positional embeddings while training the sMAE model. Reconstructions are presented in Figure 6 . Without positional embeddings, the masked tokens can not be correctly reconstructed as no positional information of masked tokens is added to the sequence.  
 
 Appendix:  


Decision: This paper proposes a surface masked auto encoder and video surface masked auto encoder for pretaining cortical signals over regular icosahedral grids. 

Strengths:
+ Interesting novel extension of prior SiT work by the authors using SOTA pre-training strategies
+ New methods for cortical analysis of strong interest to community
+ Strong experiments using large public datasets, multiple tasks for evaluation
+ Well written paper

Weaknesses: 
- Lacking statistical significance analysis
- The proposed pre-training approach comes with additional computational cost 

After rebuttal, all reviewers are agreement that this is a very strong paper; I therefore also recommend accept.

Reviews: review: paper_summary: paper_summary: This study introduces the surface Masked AutoEncoder (sMAE) and video surface Masked AutoEncoder (vsMAE) to address the challenge of encoding the spatio-temporal dynamics of brain activity, hindered by individual variations in the cerebral cortex.
main_review: main_review: Nothing to add, I would like to stay on the Oral presentation.
********************************************************************************************************************************************
strength_weakness: strength_weakness: Not sure what to suggest here.
********************************************************************************************************************************************************************************
questions: questions: none
paper_summary: paper_summary: This paper proposes two novel methodologies for multivariate and spatio-temporal pre-training of cortical signals over regular icosahedral grids: (1) surface Masked AutoEncoder (sMAE) and (2) video surface Masked AutoEncoder (vsMAE). The authors are building upon their previous work, SiT. The key idea in this paper is learning robust and generalisable representations through masking and reconstructing the surface data.
main_review: main_review: The authors have thoroughly addressed each of my comments and made appropriate updates to the relevant sections of the paper. With my concerns addressed, I am pleased to increase my rating, moving towards strong acceptance.
strength_weakness: strength_weakness: The paper lacks comparisons of baseline models, focusing solely on ablation studies. Incorporating even a basic comparison, such as phenotype predictions using a group level cortical surface atlas (Schaefer et al.), could significantly enhance the understanding of the value added by this and other attention based reconstruction methods. Adding some discussion on this point could be helpful.
questions: questions: In Section 4. Experimental Methods, the sentence 'investigating whether the encodings were neurobiologically meaningful' raises a question: Does achieving improved performance necessarily indicate that the encodings are biologically plausible?

How does the phenotypic predictions from other representation learrning methods compare to (v)sMAE? For example, (not using icosahedrons, still surface data) Jung-Hoon Kim et al., Representation Learning of Resting State fMRI with Variational Autoencoder, Neuroimage (2021). 

Minor: 
Citation available for this claim, "In the present study, ico3 is used as it was found to be a good balance between patch resolution and computational requirements."?
paper_summary: paper_summary: This paper proposes surface Masked AutoEncoder (sMAE) and video surface Masked AutoEncoder (vsMAE), both extensions of Surface Vision Transformers to address the data-efficiency gap during training. These models operate on multivariate and spatio-temporal cortical signals over regular icosahedral grids. These models are pre-trained to reconstruct cortical feature maps from masked inputs and learn good latent representations for downstream tasks such as prediction of cognitive measures. 

They evaluate their model on cortical phenotype regression using data from the young adult Human Connectome Project (HCP) and developing HCP (dHCP) and UKBiobank for transfer learning. Their models improve phenotype prediction performance as compared to models trained from scratch and provide improvements in time to convergence.
main_review: main_review: The authors have addressed a majority of my comments and improved the quality of the submission based on the suggestions provided. Overall, I think this paper is quite novel and makes a significant technical contribution to the medical imaging community and I am wiling to raise my score by a point. I hope to see it presented at MIDL
strength_weakness: strength_weakness: 1. The paper reports no baseline methods apart from different version (eg. masking ratios ) of their sMAE and vsMAE apart from different sVIT models with different data used for pre-training. How do these models compare to other phenotypic prediction models used in literature? They also do not report on the data-efficiency aspect of these models as compared to sVITs, which is mentioned in the abstract as something these models address.

2. Standard deviation measures are not reported in Tables 3 and 4. Additionally, the experiments do not performance significance testing to ensure that the performance improvements are robust. Additionally, do the experiments use only one train-test-validation split or multiple?
questions: questions: Please refer to the points raised under weaknesses

Comments: comment: **Other comments:**

*Concatenation across vertices*: We apologise for the lack of clarity in the description of the concatenation process. In short, once the triangular patches of imaging features (e.g. myelin, curvature, sulcal depth, and cortical thickness maps) are extracted from the input mesh (Figure 1.A.2), the vertices that fall into the same patch are concatenated along the feature dimensions. In practice, this means that the feature values corresponding to the vertices of a given patch are concatenated across channels. This creates a patch representation of size $45 \times C$, where C is the number of channels and $45$ corresponds to the number of vertices in a patch with ico3 sampling. Then this patch representation is projected to a token embedding of dimension $D$ with a linear layer. There is no need for a particular ordering of the imaging features/channels, however, the order of imaging features/channels must be maintained such that the concatenation process is consistent during training. More details are reported in [Dahan et al 2022, MIDL]
 
*Reconstruction error*: We quantified the reconstruction quality for the (v)sMAE models for different masking ratios and reported the reconstruction loss in Table 3 and Table 4.
We would like to thank the reviewer for the time they took to review our submission and their very constructive comments. We noted the important questions raised by the reviewer and tried to address them in the following.

**Points to address:**

*Phenotypic predictions and comparison with baseline models and other approaches*: We acknowledge the reviewer’s remarks on the lack of comparisons to baseline models for phenotyping prediction. In previous work (Fawaz et al 2022) we benchmarked the performance of surface CNNs (spectral convolution, spatial convolution) against classic ML driven by regional features derived from an atlas for the prediction of scan age (PMA) and birth age (GA) in the dHCP dataset. This showed that surface deep learning frameworks outperformed classic ML, especially for data that was not aligned (native configuration). Subsequently, in (Dahan et al 2022 MIDL 2022) we showed that the SiT improved the overall performance against these surface CNNs. In this submission, we have shown that the sMAE self-supervision of the SIT further improves performance - while, importantly, the same data splits were used throughout. We agree that a full appraisal of all these results would help readers of this paper and therefore have added these results to the appendix section in Table 5, with six baseline networks (five surface CNNs and one 2D CNN). In short sMAE pre-training improves phenotyping performance against all models, notably the two most performing gDL models in [Fawaz et al 2021], Spherical Unet and MoNet, by a large margin (apart for MoNet on PMA 0.63 MAE against 0.64 MAE, but with larger std).

*Results and data split*: We apologise for overlooking the reporting of standard deviations of (v)sMAE reconstructions in Tables 3 and 4. They are now included in the tables. Given the limited time available for the rebuttal, we did not manage to run the statistical significance performance test for the many experiments reported in our submission. However, we are still working on it and will update the paper accordingly. All the phenotyping prediction experiments on the dHCP dataset, including the surface CNNs in Table 5, use the same train/val/test split as introduced in [Fawaz et al]. For the HCP dataset, the same train/val/test split is used between the different data releases (3T, 7T). We made this point clearer in Appendix A.1.

*Data efficiency*: In Figure, we report the training loss and validation errors of different SiT models: trained from scratch, following MPP pre-training and following sMAE pretraining. The training logs demonstrate the fast convergence of the finetuned models (either from MPP or sMAE) for the difficult task of GA prediction. The data efficiency was also demonstrated through the transfer learning experiment in Figure 3.b, where a sMAE pre-trained SiT model on a large dataset (e.g UKB) is less data hungry for finetuning on phenotyping prediction tasks. Indeed, with 50\% of the data, the SiT model achieved $\leq 1$ MAE for scan age prediction, i.e. a $\sim 30\%$ improvement compared to a SiT trained from scratch, with the same amount of data. We believe that this result is particularly interesting in medical settings, where datasets are usually small in size. 

Additionally, we added a section to discuss memory requirements and computational cost of the method in Appendix A.2. However, there is an obvious additional cost of pre-training deep learning models. For instance here, sMAE pretraining on 4 cortical metrics typically takes 2 hours on a standard GPU hardware with a 128 batch size. Training vsMAE takes substantially more time (around a day or 50k iterations with a batch size of 128) because the cortical signal on task fMRI data is noisier to model. However, we believe that providing the checkpoints for these models would benefit the community and the required time to finetune new models, especially on smaller datasets.
**Other comments**: 

We apologise for the lack of clarity in the claim ‘In the present study, ico3 is used as it was found to be a good balance between patch resolution and computational requirements.’. We solely meant that ico3 patching allows patching the cortical surface at higher resolution (1280 patches against 320 as in [Dahan et al 2022, MIDL]), which allows visualising fine-grained patterns of spatio-temporal reconstruction activity, while still being not too complex to process with standard hardware. We reviewed this section to improve readability (appendix A.2)

*Computational cost:* One of the advantages of the MAE framework lies in its light computational cost, as the encoder only processes the sequence of mask patches and the decoder is made shallower. We added a section to discuss memory requirements and computational cost of the method in Appendix A.2. However, there is an obvious additional cost of pre-training deep learning models. For instance here, sMAE pretraining on 4 cortical metrics typically takes 2 hours on a standard GPU hardware with a 128 batch size. Training vsMAE takes substantially more time (around a day or 50k iterations with a batch size of 128) because the cortical signal on task fMRI data is noisier to model. However, we believe that providing the checkpoints for these models, would benefit the community and the required time to finetune new models, especially on smaller datasets.
We would like to thank the reviewer for the time they took to review our submission and their very constructive comments. We are glad that the reviewer found the present problem important and our contributions valuable. We noted the important questions pointed by the reviewer and tried to address them in the following. 

**Points to address:**

*Meaningful neurobiological encodings*:  The objective of the phenotype regression tasks was to validate whether our networks were – as a minimum - encoding phenotypically relevant information. Together with the strong generalisation performance of the model reconstructions, we would argue that this suggests that the model is learning the spatio-temporal dynamics of the brain. However, we completely agree that regression performance could be confounded by residual physiological noise signals, and as pointed out by the reviewer improved performance on phenotype prediction tasks does not necessarily equate to the biological plausibility of the learned encodings. We appreciate the opportunity to clarify our stance on this important aspect of our research. We plan to edit this statement to instead say ‘we investigated whether the encodings learned by our (v)sMAE models capture information relevant to phenotypic predictions and therefore may suggest an alignment with neurobiological patterns’. Investigation of how generalisable and interpretable the model encodings are (in the context of modelling the functional task) is out of the scope of this work (and the basis of a follow-up).

*Phenotypic predictions and comparison with baseline models and other approaches*: We acknowledge the reviewer’s remarks on the lack of comparisons to baseline models for phenotyping prediction. In previous work (Fawaz et al 2022) we benchmarked the performance of surface CNNs (spectral convolution, spatial convolution) against classic ML driven by regional features derived from an atlas for the prediction of scan age (PMA) and birth age (GA) in the dHCP dataset. This showed that surface deep learning frameworks outperformed classic ML, especially for data that was not aligned (native configuration). Subsequently, in (Dahan et al 2022 MIDL 2022) we showed that the SiT improved the overall performance against these surface CNNs. In this submission, we have shown that the sMAE self-supervision of the SIT further improves performance - while, importantly, the same data splits were used throughout. We agree that a full appraisal of all these results would help readers of this paper and therefore have added these results to the appendix section in Table 5, with six baseline networks (five surface CNNs and one 2D CNN). In short sMAE pre-training improves phenotyping performance against all models, notably the two most performing gDL models in [Fawaz et al 2021], Spherical Unet and MoNet, by a large margin (apart for MoNet on PMA 0.63 MAE against 0.64 MAE, but with larger std). 

Of particular interest, we also compared our results to the performance of Projected 2D ResNet, where icospheric surface data were projected directly to a 2D image (similarly to the data representation in Jung-Hoon Kim et al. pointed out by the reviewer). Again, the finetuning of SiT models after sMAE pre-training outperforms the 2D CNN by a large margin on the two phenotyping prediction tasks (0.97/1.93 MAE vs 0.64/1.22 MAE or PMA/GA).

Regarding the performance of phenotyping prediction of the vsMAE model on task fMRI data, we could compare the 0.39 Pearson correlation obtained here, to the 0.34 Pearson correlation obtained in [Improving Phenotype Prediction using Long-Range Spatio-Temporal Dynamics of Functional Connectivity, Dahan et al 2021] with a spatio-temporal graph convolution neural network model. In [Dahan et al 2021] functional connectome and associated time series were obtained from resting-state data from the HCP dataset following Group-ICA at different node resolutions and dual-regression of associated time courses.
We would like to thank the reviewer for their careful review of our submission and their enthusiasm for our work and contributions.