Video-CT MAE: Self-supervised Video-CT Domain Adaptation for Vertebral Fracture Diagnosis

Authors: lukas  buess
marijn f stollenga
david  schinz
benedikt  wiestler
jan s kirschke
andreas  maier
nassir  navab
matthias  keicher
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 10:56:46

Abstract: Early and accurate diagnosis of vertebral body anomalies is crucial for effectively treating spinal disorders, but the manual interpretation of CT scans can be time-consuming and error-prone. While deep learning has shown promise in automating vertebral fracture detection, improving the interpretability of existing methods is crucial for building trust and ensuring reliable clinical application. Vision Transformers (ViTs) offer inherent interpretability through attention visualizations but are limited in their application to 3D medical images due to reliance on 2D image pretraining. To address this challenge, we propose a novel approach combining the benefits of transfer learning from video-pretrained models and domain adaptation with self-supervised pretraining on a task-specific but unlabeled dataset. Compared to naive transfer learning from Video MAE, our method shows improved downstream task performance by 8.3 in F1 and a training speedup of factor 2. This closes the gap between videos and medical images, allowing a ViT to learn relevant anatomical features while adapting to the task domain. We demonstrate that our framework enables ViTs to effectively detect vertebral fractures in a low data regime, outperforming CNN-based state-of-the-art methods while providing inherent interpretability. Our task adaptation approach and dataset not only improve the performance of our proposed method but also enhance existing self-supervised pretraining approaches, highlighting the benefits of task-specific self-supervised pretraining for domain adaptation. The code is publicly available.

Main paper: 
 introduction: 
 Spinal health is a critical aspect of overall well-being and quality of life. Early and accurate diagnosis of vertebral body anomalies is essential for appropriately treating spinal disorders. Osteoporotic fractures, for instance, affect up to 12% of men and women aged 50-79 years across Europe (Harvey et al., 2010) . CT has become an indispensable tool for diagnosing vertebral fractures. However, manual interpretation of CT scans can be time-consuming and subjective, potentially leading to errors and delays in diagnosis and treatment (Carberry et al., 2013) . Deep learning has already demonstrated promising results in automating the detection of vertebral fractures (Husseini et al., 2020; Engstler et al., 2022; Keicher et al., 2023) . However, these studies have also highlighted the need for interpretable methods, as understanding the decision-making process of the models is crucial for building trust and ensuring reliable clinical application. Vision Transformers (ViTs) (Dosovitskiy et al., 2020) have shown promise for this due to their inherent interpretability through attention visualizations. However, their application has been primarily limited to 2D medical images (Ch lad and Ogiela, 2023) , as they are data-hungry and often rely on initialization from models pretrained on large-scale 2D image datasets like ImageNet (Deng et al., 2009) . This limits their effectiveness in tasks involving volumetric data, such as vertebral fracture detection in CT images. A potential solution is using models pretrained on videos, which are also 3D data with spatial and temporal dimensions (Ke et al., 2023) . Video-pretrained models offer a promising solution for initializing ViTs for 3D medical image analysis, but the domain shift between videos and medical images is substantial. An alternative approach is to use self-supervised pretraining of ViTs with in-domain data, which has been shown to improve anatomical image understanding and enhance downstream task performance (Tang et al., 2022) . Surprisingly, we find that published self-supervised ViT pretraining models significantly underperform on our task compared to CNN models, which show similar performance whether randomly initialized or pretrained on CT patches using Models Genesis (Zhou et al., 2021) . While there are many publicly available CT datasets containing spine images, vertebral fractures are rare, and datasets including these annotations are few, with a high imbalance of classes. We argue that pretraining on a task-specific unlabeled dataset with self-supervised methods, even though it contains mainly healthy vertebrae, can help the model to understand the anatomy and improve performance in detecting pathologies. Therefore, we curate a task-specific pretraining dataset and propose a novel approach that combines the benefits of transfer learning and self-supervised pretraining for vertebral fracture detection in CT scans. Our main contributions are:• We propose a framework that allows Vision Transformers to effectively detect vertebral fractures in 3D CT images despite a low data regime, outperforming CNN-based methods while providing inherent interpretability through attention visualizations.• We introduce a self-supervised domain adaptation method and a new task-specific pretraining dataset to bridge the gap between video-pretrained models and medical images, enabling the learning of relevant anatomical features in the target domain.• Our thorough experimental evaluation demonstrates the effectiveness of the proposed task-specific pretraining in improving downstream task performance for both existing pretraining methods and our novel adaptation of video-based transfer learning. 
related work: 
 Vertebral Fracture Detection Deep learning-based vertebral body classification methods can be divided into 2D and 3D approaches. 2D methods analyze a single sagittal slice from the 3D vertebra volume (Husseini et al., 2020) , which is efficient and allows initialization with ImageNet weights but fails to capture important 3D structural features. More advanced 2.5D approaches aggregate information from multiple 2D slices using RNNs (Bar et al., 2017) or LSTM networks (Tomita et al., 2018) but still do not fully exploit the 3D spatial context of CT data. Recent studies predominantly utilize 3D models to leverage the comprehensive volume data (Engstler et al., 2022; Keicher et al., 2023; Chettrit et al., 2020) , with the initial application of 3D convolutions in vertebral fracture detection pioneered by Nicolaes et al. (2020) . However, these models cannot use pretrained ImageNet weights and must be trained from scratch, which is challenging given the limited labeled medical datasets.Recent studies are concentrating more on applying transformer-based techniques to classify vertebrae. Ch lad and Ogiela (2023) explored the effectiveness of transformers in identifying cervical spine fractures from single 2D slices. Similarly, Windsor et al. (2022) employed a hybrid approach, combining a CNN for 2D feature extraction from sagittal slices with transformers for feature aggregation within and across scans. However, these methods have limitations in capturing 3D structural features due to their reliance on 2D inputs.Self-supervised Learning In recent years self-supervised learning has emerged as a popular approach for pretraining deep learning models. A recent trend in computer vision is transformer-based masked image modeling approaches, which have been inspired by the success of masked language modeling in NLP as demonstrated by BERT (Devlin et al., 2018) . Masked image modeling has already been well-established in computer vision for 2D images (He et al., 2022) , videos (Feichtenhofer et al., 2022) and multimodal models (Girdhar et al., 2023) . More recent methods like MSN (Assran et al., 2022) and I-JEPA (Assran et al., 2023) improve efficiency by using joint-embedding architectures, avoiding pixel reconstruction unlike traditional approaches.Most famous self-supervised learning methods were initially designed for natural images, but there have been significant advancements in the medical field as well. One noteworthy contribution is Models Genesis, an approach that involves pretraining a CNN-based model (Zhou et al., 2021) . Two other prominent methods, both rooted in transformer-based techniques, leverage the fusion of multiple pretext tasks such as image restoration, contrastive learning, and image rotation prediction (Tang et al., 2022) .Current self-supervised pretraining methods face two key challenges: Using non-medical data offers less domain-specificity for tasks like CT-based vertebral fracture detection, while medical-specific self-supervised learning is limited by smaller datasets compared to the natural image and video domains.Video Pretraining for CT Analysis Addressing the issue of deep learning's reliance on large labeled datasets like ImageNet (Deng et al., 2009) in medical applications, recent research has underscored the benefits of leveraging extensive video datasets for pretraining 3D medical models (Zunair et al., 2021) . Ke et al. (2023) and Rajpurkar et al. (2020) both found that pretraining 3D medical models on large-scale, out-of-domain video datasets yields better performance than training from scratch or using conventional in-domain CT datasets. These models, pretrained in a different domain, encounter drawbacks when applied to medical CT data without appropriate domain adaptation. 
method: 
 Our study involved creating a dataset comprising 27,776 unlabeled vertebra crops (refer to section Experimental Setup 4 for more details). However, it's noteworthy that our dataset's volume is still small when compared to extensive datasets like ImageNet (Deng et al., 2009) and Kinetics-700 (Carreira et al., 2019) . Transformer-based models typically necessitate training on expansive datasets (Li et al., 2023) , prompting our approach to leverage transfer learning already in the pretraining stage. Specifically, we explore using pretrained weights from data-rich domains to boost self-supervised pretraining for our vertebra CT data. Initializing CT models with ImageNet weights appears suboptimal, as they lack the capacity to capture crucial 3D details present in CT volumes but absent in 2D images. Therefore, we are focusing on using weights pretrained in the video domain. Videos, embodying 3D spatiotemporal data, present a closer alignment with the characteristics of CT volumes, potentially offering a more effective foundation for our research.Our video-CT domain adaptation method comprises three steps (see Figure 1 ). First, we employ weights from self-supervised pretraining on the Kinetics-700 video dataset, using video MAE pretraining (Feichtenhofer et al., 2022) , to establish our foundation model. The second, domain adaptation step, involves adapting these weights for vertebra CT data by pretraining on an unlabeled vertebra dataset, enhancing model alignment with the CT domain. Finally, we apply these adapted weights to vertebra fracture classification, finetuning the encoder with a labeled dataset. Video MAE for CT Vertebra Data Our approach features two self-supervised pretraining stages, both leveraging the video MAE method (Feichtenhofer et al., 2022) . To align the original 96 × 96 × 96 dimensions of vertebra CT volumes with the 16 × 224 × 224 video format from the Kinetics-700 dataset, we select 16 equidistant sagittal slices from the vertebra volume. We use sagittal slices due to their diagnostic relevance in vertebral fracture detection, as they provide crucial information for accurate assessments in this context.Positional Encoding Cropping After matching the temporal dimension of the video format, the 3D convolutional patch embedding layer (2×16×16) from video pretraining can be reused in the vertebra CT pretraining stage. However, with different input sizes (224×224 frames for video and 96 × 96 slices for CT), input token counts differ (video: 8 × 14 × 14 tokens, CT: 8×6×6 tokens), which results in a shape mismatch of the positional encodings.To address this, we introduce "positional encoding cropping", preserving only central 96×96 pixel positional encodings from videos, and discarding outer encodings (Figure 1 red ). This allows for the direct initialization of positional encoding weights in domain-specific vertebra CT pretraining using video weights, similar to the approach presented by Kim et al. (2023) .Slice Sampling During vertebra pretraining and downstream task finetuning, we employ different sampling strategies for training and evaluation phases. In the training phase, we sample 16 uniformly distributed sagittal slices. Each training step involves applying a consistent random shift to the chosen slice indices, thereby increasing diversity and effectively serving as an augmentation technique. During the evaluation phase, we maintain our initial sampling strategy but do not shift the selected slice indices. Instead, we use all six possible 16 × 96 × 96 samples covering the entire vertebral volume for prediction. This approach allows aggregation of the prediction by averaging over all sampled subvolumes, ensuring a more robust and representative result. This approach draws inspiration from multi-view testing in the video domain (Feichtenhofer et al., 2019) . 
experimental setup: 
 Preprocessing Pipeline We developed a preprocessing pipeline designed for vertebralevel classification tasks in CT scans. Initially, a spine segmentation model is applied to identify individual vertebrae. Utilizing the segmentation mask, we then extract 96 × 96 × 96 crops centered on the vertebra of interest.1 The preprocessing pipeline is illustrated in greater detail in Figure 2 and Figure 3 .In our research, we do task-specific selfsupervised domain adaptation pretraining by utilizing a big unlabeled vertebra dataset. This dataset was created by collecting seven publicly available CT datasets, each containing spine segments, inspired by the dataset selection in the CTSpine1K dataset (Deng et al., 2021) . We subsequently employed the preprocessing pipeline to process these datasets. This approach resulted in a dataset comprising 27,776 individual unlabeled vertebrae extracted from 3,446 different CT volumes. It's essential to note that this preprocessing step relies on a segmentation model, making the process unsupervised. Detailed information about the unlabeled pretraining dataset is summarized in Table 1 .  
ablation study: 
 This section details our ablation study results, analyzing video transfer learning and selfsupervised domain adaptation's impact on the vertebra classification downstream task. Table 2 outlines the performance of the key components in our method. The removal of video domain transfer learning significantly reduces performance. This emphasizes the importance of using video domain pretrained weights for our domain-specific vertebra CT pretraining, ensuring a solid foundation and enabling effective transfer of learned features from the video domain to the CT domain.The results demonstrate that video pretraining can be effectively adapted to the CT domain with proper format adjustments, even without domain adaptation. Video pretraining alone outperforms vertebra-only pretraining, aligning with the findings of Ke et al. (2023) and Rajpurkar et al. (2020) . However, skipping vertebra pretraining domain adaptation leads to lower performance than the full Video-CT MAE pipeline.Multi-View Sampling Additionally, multi-view sampling boosts prediction robustness.Combining different views of the vertebra leads to a more reliable final prediction.Positional Encoding Cropping One can see that by randomly initializing the positional encodings, the performance closely aligns with that of our full Video-CT MAE method. Yet, a closer analysis of the pretraining loss curves offers a significant insight. Employing positional encoding cropping enables a reduction in training epochs (see Figure 4 ). Another finding is that the direct use of video weights for the downstream task significantly benefits from the positional encoding cropping, as demonstrated in ablation experiment 6.Vertebra Format Adaptation Using the video model with its original 224 × 224 frame size from pretraining yields performance comparable to our full Video-CT MAE method. However, this approach results in increased training time for pretraining and finetuning.In this section, we conduct a comparative analysis between our Video-CT MAE approach and established self-supervised pretraining methods. Our focus centers on the application of these methods to our vertebra data setup, namely: Models Genesis (Zhou et al., 2021) , ViT UNETR (Tang et al., 2022) , Swin UNETR (Tang et al., 2022) , and MAE (He et al., 2022) .We study the importance of task-specific pretraining by: First, random initialization for downstream task finetuning; second, finetuning with publicly available weights; and finally, task-specific self-supervised pretraining using the public weights for initialization. Models leveraging both public and task-specific pretraining consistently surpassed those limited to public data pretraining or no pretraining at all. This was particularly evident in transformer-based models (ViT UNETR, Swin UNETR, MAE, and our Video-CT MAE), underscoring the critical role of pretraining in these architectures. Models Genesis demonstrated strong performance without pretraining, suggesting that CNN-based models may be less reliant on extensive pretraining for smaller datasets. Our Video-CT MAE method proved to be the most effective, surpassing all other evaluated methods across all metrics.We evaluate our method for vertebral fracture detection by comparing it with an existing technique and common 3D classification architectures in medical settings. In addition, we show the challenges of training 3D ViTs from scratch, which inspired our proposed approach.Method F1 (%) ACC (%) AUC (%) AP (%) DenseNet121 (Huang et al., 2017) 73.7 92.1 92.1 81.1 DenseNet169 (Huang et al., 2017) 72.9 91.7 93.3 83.9 ResNet18 (He et al., 2016) 81 Our approach significantly outperforms conventional classification architectures and shows superior results to the vertebral fracture classification method of Engstler et al. (2022) . Our presented method successfully implements 3D ViTs in a challenging 3D medical context, characterized by the scarce and imbalanced labeled data. 
conclusion: 
 Our novel approach adapts video-based transfer learning through task-specific self-supervised domain adaptation, successfully enabling Vision Transformers to address vertebral fracture detection in 3D CT scans. This study not only advances the state-of-the-art in vertebral fracture detection but also showcases the potential of task-specific pretraining for other medical image analysis tasks. Future research could explore the creation of task-specific pretraining datasets for various applications and evaluate the generalizability of our approach. By addressing the challenge of limited data in medical image analysis, our work offers a promising solution for improving patient care through accurate, interpretable, yet resource-efficient methods for advancing clinical decision-making support. (Dosovitskiy et al., 2020) as encoder and a decoder depth of 4. We use convolutional patch embeddings of size 2 × 16 × 16. A masking ratio of 0.8 is applied. We initialize domain-specific vertebra CT pretraining with weights from Kinetics-700 pretraining, utilizing positional encoding cropping. Training consists of 100 epochs with a batch size of 8, using the AdamW optimizer and a cosine learning rate scheduler (base learning rate 1e-3). Data preprocessing involves clipping HU values to the range -1000 to 1000 and then scaling them to 0-1. To match the RGB video format, we replicate the HU values. The input volumes have a shape of 16 × 96 × 96, following the described sampling technique. 
 
 Appendix: The MAE pretraining architecture is adapted for the subsequent classification task, exclusively utilizing the ViT-Large encoder. Global average pooling is applied to the output tokens, followed by a linear classification layer for binary classification. Finetuning is done for 50 epochs, employing a batch size of 64, Adam optimizer, and a cosine learning rate scheduler with warmup (base learning rate 1e-5). Class weighting is introduced to the cross entropy loss to handle data class imbalance. Data preprocessing remains consistent with pretraining. The labeled vertebra dataset is split into training, validation, and testing sets at a 60%/20%/20% ratio, ensuring patient-level separation to prevent test-leakage.Segmentation model is provided by the ImFusion GmbH (approach similar to Bürgin et al. (2023))Finetuning Minutes/Epoch 



Reviews: review: paper_summary: paper_summary: Authors propose to use self-supervised fine-tuning of a
transformer-based architecture that was pre-trained using videos,
before fine-tuning for task. They created a large data sets of
volumetric CT crops over vertebrae and fine-tuned a ViT-Large using
the same strategy as the original pre-training with videos, but using
vertebrae crops. Some adaptations were necessary. This self-supervised
transfer of the original ViT proved to be useful in ablation studies
and comparisons with other architectures.
main_review: main_review: I am happy with the revisions authors made to the article. This improves the quality of the article by a large margin. 
However, it also means they applied a lot of change to the article. If this is acceptable, I recommend accepting the article. If this is an issue, I believe the article should not be accepted in this cycle and authors should be encouraged to submit to the following conference.
strength_weakness: strength_weakness: - The introduction and the abstract can be sharpened to more clearly
  state the main contribution of the work. Especially, the abstract is
  a bit cryptic. I believe sharpening the text would greatly
  facilitate the dissemination of the work. Effectively, the paper
  proposes to use self-supervised transfer learning for a specific task.
- Technical novelty of this work remains on the lower end.
- The transfer self-supervised learning is quite task
  specific. Authors perform this training on volumetric CT-crops
  around vertebrae. It is unclear, how much this can be extended to
  other CT-related tasks.
- While the ablation studies are great, the comparisons with existing
  techniques for the same task is not very informative. To the best of
  my understanding, authors do not compare with previous work on
  fracture detection. Since the paper is largely an empirical study, I
  strongly believe a proper comparison is necessary. This would allow
  readers to compare the pre-training strategies with existing models
  for fracture detection. I believe comparisons are simply different
  generic architectures.
questions: questions: - For the rebuttal, it would be important to do comparisons with
  existing techniques for fracture detection. Otherwise, it is not
  possible to understand the value of the proposed technique.
- It would also be interesting to demonstrate performance in the case
  where the self-supervised pre-training is not restricted to
  vertebrae crops but performed on the entire CT volumes. This would
  explain whether the methodology can create pre-trained networks that
  can be used for other CT-related tasks.
paper_summary: paper_summary: This paper introduces improving vertebral fracture diagnosis through self-supervised learning, utilizing a unique video-CT domain adaptation technique. Comparative analysis reveals that the proposed approach outperforms existing self-supervised learning methods in downstream vertebra classification tasks.
main_review: main_review: The rebuttal has addressed most of my concerns, including novelty, the initial absence of code during the first review phase, and additional ablation experiments.
I have revised my score to Weak Accept.
strength_weakness: strength_weakness: - The provided code link contains no actual code.
- Ablation experiments should focus on the impact of Video Pretraining on methods like Models Genesis (Zhou et al., 2021), ViT UNETR (Tang et al., 2022), Swin UNETR (Tang et al., 2022), and MAE (He et al., 2022); rather than being trained on the unlabeled vertebra dataset, which is evidently a useful strategy.
- Regarding Table 2, the authors are encouraged to provide experimental results for additional combinations of Ablation Elements.
- The approach of video MAE pretraining followed by positional encoding cropping, vertebra CT pretraining, and downstream task fine-tuning seems to be quite straightforward, with limited technical innovation.
questions: questions: Concerns about limited novelty and experiments.
paper_summary: paper_summary: Authors propose an approach to solve the task of binary classifying fractures on CT images of vertebra. The approach is based on three stages, (1) transfer learning from video ViT model (2) pretraining with MAE on a large unlabeled dataset (3) fine-tuning on the desired task with labeled data. The authors also use cropped positional embeddings and slice sampling to adapt the new task input size to the pretrained model. Ablation experiments are provided regarding the different stages and proposed modules of the method, as well as comparisons to other relevant transfer learning and pretraining approaches.
main_review: main_review: During the rebuttal period, authors provided a supplementary relevant comparison method on vertebra classification, as well as a few clarifications on the introduction and results analysis. 
However, as mentioned by all reviewers, this manuscript provides an interesting message on pretraining and self-supervised learning (on a specific dataset), but overall the technical novelty and study impact remain limited. 
So I would keep my rating to borderline, on the accept side.
strength_weakness: strength_weakness: - either lack of novelty (I deem it is quite common to use (1) transfer learning from natural images or video models (2) pretraining with some self-supervised method such as MAE on large unlabeled data (3) fine-tuning on the desired task with labeled data stages) or lack of evidence that such approach could be more broadly used for medical datasets (only one task solved); Especially regarding the fact that authors claim to work under small data regime but they successfully curated 27k samples.
- lack of comparison methods regarding studies solving the same task (only comparison methods to similar methodologies, not initially proposed to solve medical-related tasks), comparison expected at least from the state-of-the-art presented in the related work;
questions: questions: - Comparison to at least the SOTA method to solve this specific task;
- Clarify, specify contributions;
- Add at least one dataset to demonstrate the relevance of the proposed method to other medical applications (and, to me, accounting for the rather weak methodological novelty);
- Answer questions in the major comments.

Comments: title: Response to rebbutal
Revised submission: additional experiments; vertebral fracture detection SOTA comparison; more precise contributions and motivation of proposed method
Revised submission: additional experiments; two new ablation experiments; more precise contributions and motivation of the proposed method
Revised submission: additional experiments; vertebral fracture detection SOTA comparison; more precise contributions
comment: Thank you for your detailed answer and modifications to the manuscript.
## Comment:
Thank you for your comprehensive review and the time you dedicated to understanding our work. Your insightful comments and constructive suggestions are greatly appreciated, as they have helped us refine and improve the quality of our paper. Based on your feedback, we have made substantial revisions to the manuscript, with a particular focus on enhancing the experimental design and improving the overall clarity of the paper. We believe these changes have strengthened the work and made it more impactful.

## Questions to address:
We appreciate the reviewer's emphasis on the necessity of benchmarking our method against established techniques in vertebral fracture detection to position our work in the community. To address this, the revised version of our paper includes a comprehensive comparison table. Table 4 contrasts our approach with a state-of-the-art method in vertebral fracture detection and commonly used 3D classification architectures within the medical field. Furthermore, we have incorporated details of our preliminary experiments using ViTs, highlighting the challenges and our motivation behind the proposed method for implementing ViTs (with their inherent interpretability ability) in our challenging task.

We would like to thank the reviewer for bringing our attention to the initially unclear definition of the paper's contributions. In the revised version, we have improved both the abstract and the introduction to more clearly and precisely articulate the paper's motivation and key contributions. We highlight that our approach, which uniquely combines existing methods, represents a novel contribution. Specifically, this is the first method, to our knowledge, applying transfer learning from the video domain to initialize task-specific, self-supervised pretraining and not directly initialize the downstream task. This approach enables 3D ViT-based models to effectively handle complex 3D medical data in scenarios with limited labeled information.

The reviewer's suggestion to evaluate our approach's generalizability across different medical applications is interesting. Our current focus is on underscoring the significance of task-specific pretraining. Regrettably, we could not apply our method to a different medical application with a similar data setup, allowing task-specific pretraining. Nevertheless, our paper lays the groundwork for future research, potentially paving the way for developing a generic pretrained model for full-body CT scans.

## Other concerns:
Addressing the initial absence of code during the first review phase, we have now published it. The code is accessible through the link provided in the paper's abstract.

We recognize the validity of the reviewer's suggestion to publish our dataset. Unfortunately, licensing restrictions on the original datasets prevent us from releasing the vertebrae crops. However, we can make the code used to create the dataset available. Other researchers can replicate the dataset using this code, together with the vertebral segmentation model from ImFusion GmbH. Detailed information on the preprocessing pipeline is included in the appendix of our paper. We also added a citation to a relevant paper by ImFusion GmbH on vertebra localization.

In the comparison of ImageNet and video weights, our downstream task analysis revealed that ImageNet weights did not surpass random initialization in our data setup. Consequently, this insight led us to favor video data pretrained weights, demonstrating superior performance.

To clarify our dataset setup: We curated 27K unlabeled vertebrae crops with no fracture label from public datasets originating from 3,500 different patients. We have 6K labeled vertebra crops, including 940 fractured ones, from 457 distinct patients. We think it is valid to say that we work under a small data regime.

We want to thank the reviewer for highlighting the reference on positional encoding cropping, of which we were unaware. We've now included this citation in our paper.

We've updated the paper to clarify our sampling scheme. In the training phase, we sample 16 uniformly distributed sagittal slices. Each training step involves applying a consistent random shift to the chosen slice indices. During the evaluation phase, we maintain our initial sampling strategy but do not shift the selected slice indices. Instead, we use all six possible 16x96x96 samples covering the entire vertebral volume for prediction.
Our sampling method, we believe, is more effective than sliding window as it ensures the entire volume is represented in each sample, whereas sliding window may result in only a few windows containing the fracture.
We haven't explored 96x96x96 training due to necessary additional changes in the patch embedding layer. Future work may consider positional encoding inter/extrapolation in combination with weight inflation in the patch embedding layer to make 96x96x96 with 16x16x16 input patches possible.
## Comment:
We greatly appreciate the feedback and valuable recommendations provided by the reviewer. We aim to ensure that our paper is a useful and accessible resource for the entire MIDL community and provides a strong foundation for future studies. In response to the comments received, significant improvements have been made, with particular attention paid to refining the experiments for ablation studies and task-specific pretraining. We additionally added experiments to position our work in the community of vertebral fracture detection.

## Questions to address:
We appreciate the reviewer's emphasis on the need for comprehensive experiments. In response, we have enriched our study with additional experiments, particularly focusing on the impact of task-specific pretraining. Our expanded self-supervised pretraining methods comparison now clearly illustrates that task-specific dataset pretraining enhances performance compared to using public weights or training from scratch.

To further illustrate the impact of our research on vertebral fracture detection, the revised paper includes new experiments. In Table 4, we introduce a comparison that contrasts our method with a state-of-the-art approach for vertebral fracture detection and other established 3D classification architectures for medical data. Furthermore, we have incorporated details of our preliminary experiments using ViTs, highlighting the challenges and our motivation behind the proposed method for implementing ViTs. These additions underscore our method's significant advancements over traditional classification architectures and its superior performance compared to a state-of-the-art model in vertebra fracture classification. Our analysis of the ViT models demonstrates that we successfully implemented a method that makes 3D ViTs effective in the challenging 3D medical context, overcoming the hurdles of limited and imbalanced labeled data.

We appreciate the reviewer's perspective on the perceived novelty of our method. We would like to highlight that our approach, which uniquely combines existing methods, represents a novel contribution. Specifically, this is the first method, to our knowledge, applying transfer learning from the video domain to initialize task-specific, self-supervised pretraining. This approach enables 3D ViT-based models to effectively handle complex 3D medical data in scenarios with limited labeled information. We believe our method lays a significant groundwork for future explorations in this area.

## Other concerns:
Addressing the initial absence of code during the first review phase, we have now published it. The code is accessible through the link provided in the paper's abstract.

We find the suggestion of pretraining other methods on video datasets interesting. However, the aim of our work is to demonstrate the adaptation of publicly available weights to task-specific datasets. This approach allows effective pretraining with limited computational resources, bypassing the extensive time and hardware requirements of pretraining on large datasets such as Kinetics-700. Indeed, the proposed idea holds merit and presents a valuable direction for future research.

In response to the suggestion for additional ablation experiments, we have included two new ablation experiments. The first experiment (Experiment 5) compares the performance of domain-specific pretraining and downstream task finetuning using the original 16x224x224 video format versus the adjusted 16x96x96 format. Although the results are similar, we observed a significant increase in training time with the original video format. The second experiment highlights the crucial role of positional encoding cropping when directly transitioning from video pretraining to the downstream task, bypassing the task-specific pretraining stage.

Addressing the comment on the comparable performance depicted in the loss curves between random positional encoding initialization and positional encoding cropping, we have refined our discussion in the text. We argue that despite the similar performance, positional encoding cropping offers a distinct advantage by facilitating faster convergence. This efficiency leads to a reduction in the number of training epochs required, consequently saving valuable training time. The reason for this is that the positional encodings can be relearned during the self-supervised training. When not relearning the positional encodings using the self-supervised domain adaptation step, the results on the downstream task get worse (see ablation experiment 6).
## Comment:
We sincerely thank the reviewer for the insightful comments and constructive suggestions regarding our paper. We want the paper to be accessible and useful for the entire MIDL community as we think the paper sets a good foundation for future research. In response to the feedback received, we have undertaken substantial revisions, particularly focusing on enhancing the experimental design of our paper.

## Questions to address:
We appreciate the reviewer's emphasis on the necessity of benchmarking our method against established techniques in vertebral fracture detection to position our work in the community. To address this, the revised version of our paper includes a comprehensive comparison. The results added in Table 4 contrast our approach with both a state-of-the-art method in vertebral fracture detection and commonly used 3D classification architectures within the medical field. Furthermore, we have incorporated details of our preliminary experiments using ViTs, highlighting the challenges and our motivation behind the proposed method for training ViTs on challenging 3D tasks with limited data. These additions underscore our method's improvement over established classification architectures and its superior performance compared to the state-of-the-art in vertebra fracture classification.

The reviewer's suggestion about evaluating generalizability through pretraining on the entire CT volume (not only vertebra crops) is interesting, and we compare it to Model Genesis pretrained weights that followed this approach. However, within the scope of this paper, we focused on improving a challenging downstream task as opposed to a general feature extractor for CT images. We show experimentally that this task-specific pretraining with unlabelled data is highly effective in improving the downstream task performance as highlighted in the now extended Table 3. We agree that for future research it would be interesting to investigate the generalizability of this approach to other medical applications.

## Other concerns:
Thank you for highlighting the need for improved clarity in our abstract and introduction. In response, we have made extensive revisions in both sections of the revised version of our submission. These improvements aim to clarify the motivation behind our work and to explicitly outline the main contributions.

Furthermore, we have expanded our experimental section to more comprehensively demonstrate the impact of task-specific pretraining. These additional experiments robustly illustrate that pretraining on a task-specific dataset notably enhances performance compared to utilizing publicly available weights or training from scratch.

We appreciate the reviewer's perspective on the perceived novelty of our method. We would like to highlight that our approach, which uniquely combines existing methods, represents a novel contribution. Specifically, this is the first method, to our knowledge, applying transfer learning from the video domain to initialize task-specific self-supervised pretraining. This approach enables 3D ViT-based models to effectively handle complex 3D medical data in scenarios with limited labeled information. This contribution is evident in the gain in performance over existing self-supervised pretraining methods for medical 3D data and naive transfer learning from video weights shown in Table 3. We believe our method lays a significant groundwork for future explorations in this area.