Modeling the acquisition shift between axial and sagittal MRI for diffusion superresolution to enable axial spine segmentation

Authors: robert  graf
hendrik  möller
julian  mcginnis
sebastian  rühling
maren  weihrauch
matan  atad
suprosanna  shit
mark  mühlau
johannes c paetzold
daniel  rueckert
jan s kirschke
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 10:49:36

Abstract: Spine MRIs are usually acquired in highly anisotropic 2D axial or sagittal slices. Vertebra structures are not fully resolved in these images, and multi-image superresolution by aligning scans to pair them is difficult due to partial volume effects and inter-vertebral movement during acquisition. Hence, we propose an unpaired inpainting superresolution algorithm that extrapolates the missing spine structures. We generate synthetic training pairs by multiple degradation functions that model the data shift and acquisition errors between sagittal slices and sagittal views of axial images. Our method employs modeling of the k-space point spread function and the interslice gap. Further, we imitate different MR acquisition challenges like histogram shifts, bias fields, interlace movement artifacts, Gaussian noise, and blur. This enables the training of diffusion-based superresolution models on scaling factors larger than 6$\times$ without real paired data. The low z-resolution in axial images prevents existing approaches from separating individual vertebrae instances. By applying this superresolution model to the z-dimension, we can generate images that allow a pre-trained segmentation model to distinguish between vertebrae and enable automatic segmentation and processing of axial images. We experimentally benchmark our method and show that diffusion-based superresolution outperforms state-of-the-art super-resolution models.

Main paper: 
 introduction: 
 Magnetic Resonance Imaging (MRI) has become a cornerstone of medical imaging. While improvements in MRI scanners have enabled widespread clinical adaptation, imaging requires a careful balance of acquisition time, required scan resolution, and signal quality (Plenge et al., 2012) . To reduce scan time, medical imaging research has developed a variety of super-resolution (SR) algorithms that aim to reconstruct high-resolution from low-resolution scans featuring sharp and anatomically faithful details whilst ideally being resistant to hallucinations. While advancements have led to noticeable achievements in various MRI applications, improving spinal multiple sclerosis (MS) imaging is particularly challenging. In MS diagnosis, precise imaging is required, as detecting MS lesions is inherent to diagnosing and monitoring the disease (Wattjes et al., 2021; Kearney et al., 2015) . Registration and structure/MS segmentation are well established in brain imaging, while spinal imaging has been trailing behind due to its more challenging acquisition and analysis process. Only recently, it gained traction within the MS community since guidelines have shifted a focus to spinal cord lesions (Wattjes et al., 2021) and recent technical advances such as the development of the Spinal Cord Toolbox (SCT) (Leener et al., 2017) . In clinical routine, T2-weighted (T2w) MRI is crucial in assessing lesion load and disease activity. Due to the size of the spine, imaging is typically performed using 2D sagittal or axial views with relatively high slice thicknesses, typically ranging between 3-10 mm. This poses an inherent challenge to 3D segmentation, as axial 2D views frequently lack the intricate details and context needed to accurately segment vertebral structures and intervertebral discs. At the same time, axial scans are preferred in the context of lesion activity. To conduct MS studies for the spinal cord, it is necessary to register spine scans to a common template like the PAM50 atlas (De Leener et al., 2018) . Current registration tools (Leener et al., 2017) require manual vertebra-level labels to function. Although whole spine segmentation for sagittal images was made available recently (Graf et al., 2023; Möller et al., 2024) , the problem of axial vertebra instance segmentation persists due to the challenges posed by large slice thicknesses. Through super-resolution, we can enable segmentation on axial images. With such a segmentation, we could automatically extract the vertebra-level label, which can be utilized for atlas registration and enables large-scale studies to access MS lesion growth in the spinal cord. This paper proposes a solution to these challenges by enhancing the quality of axial images for segmentation to the same resolution as the inplane resolution of a sagittal slice. The approach involves using sagittal images only for training, avoiding the difficulties associated with aligning sagittal and axial images. We improve the existing degradation model of SMORE (Zhao et al., 2021) , based on MRI scan properties, to learn the impact of shifts in distribution between axial and sagittal images. We change the zero-shot-learning approach of SMORE to an unpaired sagittal plane of axial to sagittal super-resolution. We compare existing super-resolution networks with the conditional diffusion models Palette. Our pipeline is the first that can automatically segment full vertebrae in axial T2w images. We achieve a realistic-looking >6x super-resolution without paired or isotropic data. 
related work: 
 Superresolution. Superresolution (SR) enhances image sharpness and details. Various models have been introduced to upscale paired low-resolution (LR) images to high-resolution (HR) images (Bashir et al., 2021; Bhowmik et al., 2017) . In the context of MRI, isotropic HR images are often downscaled to LR images through downsampling for training (Feng et al., 2022; Chen et al., 2018; Li et al., 2022) . Subsequently, a deep learning model, such as a GAN (Zhou et al., 2022) or regression model, learns the inverse process. More recent papers investigate denoising diffusion as an SR model (Chung et al., 2023; Wu et al., 2023; Kawar et al., 2022) . Some SR methods focus on enhancing LR patches to HR patches (Mithra et al., 2021; Zhao et al., 2021) . For instance, SMORE (Zhao et al., 2021) utilizes zeroshot-learning, where high-resolution plane patches are degraded with an MRI-specific point spread function. The orthogonal LR plane is then upsampled and super-resolved by the model. However, a limitation of patch-based SR is its inability to introduce features that are larger than a patch or perform biologically informed slice inpainting, which is crucial for our task because we want to reintroduce the z-axis information to distinguish vertebra instances. The SMORE zero-shot approach can not learn how a vertebra looks under different rotations because it has never seen it, especially given the large-scale factors. Other papers explore jointly modeling axial and sagittal images (Liu et al., 2021; McGinnis et al., 2023; Zhao et al., 2019) , aiming to enhance correctness. Nevertheless, these approaches necessitate registration to align these images, posing a "chicken and egg" problem for our task.Spine segmentation. Vertebra segmentation is well-established for CT through the VerSe Challenge (Sekuboyina et al., 2021) and its participants (Payer et al., 2020; Chen et al., 2020) . Regarding sagittal T2w MRI, two datasets are available for the lumbar spine-Spider (van der Graaf et al., 2023) and MRSpineSeg Challenge (Pang et al., 2020) . Li et al. (2021) developed a 2D slice-wise semantic segmentation for axial images. Recent advancements have extended CT whole spine segmentation to sagittal MR images (Graf et al., 2023) . Möller et al. (2024) leveraged these labels and incorporated intervertebral disc and spinal channel segmentation from Streckenbach et al. (2022) . We employ their publicly available segmentation algorithm as a downstream task on our super-resolved axial images to generate segmentation. On native axial resolution, this model can not untangle vertebra instances due to the strong resolution mismatch.Registration. The Spinal Cord Toolbox (Leener et al., 2017) facilitates registration to an atlas when spinal cord segmentation and vertebra height points are accessible. However, automatic vertebra height extraction proves unreliable on thick axial images and necessitates manual annotation. In contrast, our segmentation on axial SR can autonomously generate these points, offering a fully automatic solution. 
methodology: 
 Problem statement Given two types of slice-wise MRI with different high-resolution planes, we want to upscale a low-resolution view to the same resolution as the other image. For this, we aimed to investigate degradation functions and a diffusion-based SR algorithm. Pairing real data is infeasible due to the thick sliced spine images suffering from intrinsic motion, missing isotropic ground truth data, and misaligned sagittal volumes. To assess the quality of the SR data, we assessed spine segmentation as a downstream task. The segmentation enables labeling and registration to a common atlas to evaluate multiple sclerosis statistics in the axial images in future studies, a task previously not solved.Dataset We have an internal dataset with 416 multiple sclerosis patients (split 336/45/36) with multiple follow-up sessions (total=927/136/93). Each session contains ≈ 2 -4 axial (total=2742/404/282) and ≈ 2 sagittal T2w MRI scans (total=1883/285/193). Our data exhibits a through-plane spacing of 3-4 mm in sagittal images and 5 mm in axial images, representing a standard slice thickness for such scans. See Figure 1 on how the resolution differs between sagittal views of an axial and a sagittal image. Due to the large partial volume effects and inter-vertebral movement, we could not register the axial and sagittal image beyond the automatic alignment with an error of up to 30 mm (Figure 7 ).A degradation function is the forward mapping from an HR to an LR image. A naive approach would be interpolation and random Gaussian noise (Bashir et al., 2021) . For MRI images, this approach does not correctly model the signal acquisition.Point spread function (PSF) MRIs are acquired in k-space, and the PSF must be modeled in k-space instead of the image space. Inspired by SMORE (Zhao et al., 2021) , we use a convolutional kernel to degrade the 2D HR image that simulates the partial volume effect caused by the k-space PSF. While SMORE only downscales the image to LR, we both down-and upscale the image to the same pixel space as the HR image. We add random Gaussian noise in between the up-and downsampling process. The target sagittal in-plane resolution is the same as the pretrainend segmentation network with 0.8571 mm. We work with clinical data and expect that we cannot rely on the resolution being always exactly 5 mm. With the down-and upscale steps, we are not limited to any fixed integer scaling factor. Instead of one fixed kernel, we take multiple PSF kernels with starting resolutions between 4 and 9 mm in 0.25 mm steps.Interslice gap randomization The signal of a 2D slice is measured with a target thickness. The thickness of this volume is not the slice distance (Bradley and Glenn, 1987; Schwaighofer et al., 1989) . We found the measured thickness to be 4 mm. This volume is not constantly integrated but must drop from the center focus to the 2 mm edges to the side; otherwise, we would observe very blurry images. As we only have discrete pixel rows, we take up to half (1 to n slices chosen randomly) of them inside a slice thickness and weighted sum them together. We repeat this generated pixel row for the full slice thickness. This process simulates a random volume effect rather than the real partial volume effect and intentionally loses information about slices between two axial image slices. This process is done before the downsampling of the SMORE degradation.Histogram MRI images have no fixed values but are instead images between 0 and a max-value. We linearly rescaled the image to [0,1]. The histogram changes between scans and MRI devices. We used color jitter to randomize the histogram (0.8 -1.2 factor). This should make the model histogram shift agnostic. We tried manual parametric histogram shifts, but they added nothing to the performance when we used them, along with the more general color jitter.Interlace movements artifact Slice interleaving is the process of scanning every other slice first to prevent cross-talk between slices. If the subject moved during the image acquisition, we observe that every other slice is shifted in the front-back direction. We simulate this in 15% of the LR images by moving an interlace pattern with 4 to 10 pixels thickness by 0.2% to 2% of the image width to the back. See Figure 4 for an example.Others We add other common image degradations like random MRI-bias field (Van Leemput et al., 1999; Sudre et al., 2017) , Gaussian noise, and blur. We do a random rescale (factor 0.75 to 1.5) and randomly rotate up to 5 degrees. PSF and interslice gab simulation are applied all the time. All other degradation have a chance of 30% to be applied. For the order of operation, see Figure 2 . We pad and crop the images to 160 by 160, which is roughly the size of a sagittal view of an axial image.Our chosen SR-model is Palette, a diffusion image to image model (Saharia et al., 2022) . Palette is a diffusion model for conditional image translation, inpainting, and image reconstruction. It use the denoising diffusion implicit model (Song et al., 2020; Ho et al., 2020) , which is conditioned by concatenating the LR image to the noised HR image as an input. The model is a 4-block UNet. It uses SiLu activation and 32-channel GroupNorm. The timestep embedding is added by scaling and shifting the channels in every ResBlock. We only superresolve single sagittal slices. The slice superresolution is mostly consistent, but some image artifacts are visible in the axial view. Like Zhao et al. (2021) , we could introduce a second phase that removes the created artifact from the axial view by superresolving an axial image and producing a paired data set of the original image and its corresponding slices. We omitted this for simplicity. 
experiments and results: 
 We manually corrected a vertebra instance segmentation ground truth for our axial test data, where at least the vertebral body is fully segmented. Predicted vertebra instances are matched with the center of mass of the vertebra body to this ground truth. We count how many vertebrae are missed by merging or omitting. A vertebra is considered omitted when no vertebra body center mass point is inside the instance-corrected ground truth mask. We analyze the vertebra components by computing the Betti error b 0 and b 1 (Stucki et al., 2023) . We expect a vertebra to be a single component b 0 = 1. A vertebra has a single hole for the spinal cord, and the neck vertebrae contain two additional holes called transverse foramina, which are openings for vessels to pass through. The used segmentation algorithm cannot reproduce the additional holes. We expect the number of holes to be b 1 ∈ [1, 2, 3] for neck vertebrae and b 1 = 1 for the rest. We count error rates per vertebrae that are missing or do not fit this description. We use this metric because we observed that the segmentation network often has issues with completing the structure around the spinal channel. We manually annotate 107 random axial slices of the bony vertebra, each in a different axial volume, by two medical experts. The sagittal segmentation algorithm segments the superresolved axial image. Then, we sample the segmentation back to the original volume and compute the Dice score for our expert annotation. Unlike other papers, we omitted PSNR or SSIM from our analysis due to the lack of a suitable ground truth. As a proxy, we use a downstream task quality to evaluate whether contrast enhancement and inpainting improve the segmentation and instance detection. Therefore, this work optimizes the structural enhancement instead of the perceptual quality of the images.We compare existing models with the Palette diffusion approach, all with our degradation function. We use cubic interpolation, GAN-based ESRGAN+RRDBNet (Wang et al., 2018; Feng et al., 2022) , regression-based RCAN (Zhang et al., 2018) and transformer-based HAT (Chen et al., 2023 ) model as baselines. We report them in Table 1 and visualize differences in Figure 3 . The diffusion model Palette was better suited than the best baseline RCAN for superresolution. RCAN performance is competitive with Palette. As already discussed by Zhang et al. (2018) , all their tested SR models drop in performance with large scaling factors. We reckon Pallete could handle larger superresolution factors better than other models. Further research is needed for confirmation.Table 1 : Quantitative comparison of downstream 3D segmentation. The axial SR is segmented by a pre-trained network. Vertebra detection rate (dtr) is the percent of vertebrae correctly identified through translation and segmentation in axial images. We reported the Betti numbers error rate (er) when a vertebra has not b 0 = 1; b 1 = 1 or ∈ [1, 2, 3] for neck vertebra. "axial Dice" is the Dice score on a binary vertebra-bone annotation with 107 slices in different axial scans evenly distributed through the subjects. We do an ablation on our degradation function. See Table 2 for exact values. The Dice score of axial slices does not improve much because of large interpolation artifacts. The segmentation network rescales the image in the left/right direction to 1.25 mm. Other common errors are structures not modeled in the sagittal segmentation, like the transverse foramina or the transverse process of the vertebra, which is outside of the field of view of most sagittal images. Our approach improves the vertical correctness of the images. The betti errors towards b 0 do not follow a trend, while b 1 shows our additional augmentations improve the segmentation around the spinal channel. Our ablation is limited to additional augmentations because the chosen SR-model plus segmentation model is quite stable, even with some SR anomalies, and we believe that we have reached the segmentation model's upper limit. We introduced additional augmentation to be invariant against movement artifacts, MR bias-field, changing through-plane resolution, and image corruption through noise or blur. The test showed that these changes do not worsen the baseline results. 
conclusion: 
 In this work, we address the previously unsolved task of full 3D vertebrae instance segmentation in clinical axial MRI images. This was unsolved because the z-resolution of clinical axial MRI images is typically larger than the intervertebral disc or substructures of the vertebrae. We achieve this by training unpaired superresolution diffusion models with a newly developed degradation function. Our degradation function is informed by real-world MRI properties such as the point spread function, the interslice gap, the interlaced acquisition, histogram shift, and bias fields. We can artificially introduce a domain shift from HR to LR images and create aligned training data. We do not require isometric data nor aligned axial and sagittal images. Our diffusion model can inverse this process by training on the generated pairs. During inference, we only need the real axial images to superresolve them. SR for large scaling factors is strongly ill-posed, and our model is limited to a best guess. The structures between slices, as expected, vary from the real structure. The superresolution enables us to segment the axial image and compute points for registration; both were previously not possible. See Figure 5 and 6. In the future, our approach can be extended to any other MR acquisition protocol with resolution differences.Figure 5 : Examples of our inference pipeline. We stitched and cubic upscaled the axial images to isometric 0.8571 and used the sagittal plane to upscale the image. We show the segmentation in the slice and 3D render. The segmentation can be used to extract the vertebra heights for registration.  
 
 Appendix:  


Decision: The  reviewers mostly agreed that the authors addressed their concerns and found novelty in some aspects of this paper. They agreed that there is a need to perform superresolution reconstruction to enable axial spine segmentation.

Reviews: review: paper_summary: paper_summary: Authors present an image super-resolution method that focuses on
synthesizing missing slices in thick-slice axial MRI acquisitions of
the spine. Problems with segmentation and modeling with axial
acquisitions are highlighted. Alignment with sagittal acquisitions is
deemed problematic. To that end, a model that directly super-resolves
an axial acquisition in the through-plane direction is
proposed. Super-resolved images are evaluated via a segmentation. The
super-resolved images are segmented and then resampled back to the
coordinate frame of the original thick-slice axial
acquisition. Resampled segmentations are compared with ground
truth. Comparisons with different super-resolution techniques are also
presented alongside an ablation study.
main_review: main_review: I thank the authors for the lengthy explanation. While I appreciate the applications more now, I am still not convinced regarding the demonstration of the novelty. The novelty - despite what authors claim - I believe is in the degradation process. As I wrote before, Palette is not new. Therefore, results in Table 1 only shows advantages of Palette - which is prior work. Furthermore, authors mention that the registration process fails and that leads to a mismatch. Dice score is claimed to not improve. It is crucial to report this DICE score and then compare it what is obtained after the SR process. Furthermore, Figure 7 provides some examples but after a simple alignment, do they not get better? Authors can provide segmentation overlays after alignment and compare with the result of the proposed method. In summary, I still believe (1) the novelty is in degradation and its value should be demonstrated more clearly and (2) a better demonstration of the difficulty of using the sagittal scans should be provided.
strength_weakness: strength_weakness: - The work needs stronger motivation in my opinion.
  * The main claim of this article is that sagittal acquisitions
    cannot be used together with axial acquisitions. More precisely,
    registration of these two acquisitions does not work very
    well. This claim is not substantiated in this article. Examples of
    such misalignments, results of registration as well as details of
    the registration methods used should be provided.
  * The goal of the work is not very clear. Specifically, it is not
    clear whether authors are trying to replace sagittal
    acquisitions. If it is so, then stronger fidelity with the
    sagittal acquisitions need to be demonstrated. If the authors are
    not trying to replace sagittal acquisitions, then I am assuming
    one would always use the sagittal acquisitions for segmenting
    vertebrae as well as the discs for quantitative analysis. The
    axial acquisitions, as mentioned in the text, are preferred in the
    context of lesion activity. In this scenario, the need for
    segmentation of the axial acquisitions as well as why a
    segmentation obtained from the sagittal acquisitions could not be
    used for the axial acquisition should be justified.
- To the best of my understanding, the main novelty of the presented
  work is the degradation process. In terms of the learning algorithm
  the novelty is low. The conditional generation method is from a
  previous work.    
- In the experimental analysis, comparing segmentations in the
  coordinate system of the axial-acquisition may not be evaluating the
  quality of the super-resolved image directly. I believe the
  comparison should be between a segmentation obtained from a
  super-resolved image and a sagittal acquisition.
- Authors compare all conditional generation methods using the
  degradation process proposed here. Therefore, the comparisons are
  actually focused towards showing performance gains due to
  Palette. However, this is not the contribution of this paper. The
  main contribution seems to be the degradation process, and that is
  the component that should be evaluated.
questions: questions: The main question to answer is the motivation. In its current format,
the motivation for the proposed work need to be substantially
strengthened.

Furthermore, I believe directly showing the benefits of the
degradation process, rather than comparing different conditional
generation methods would better position the article.
paper_summary: paper_summary: Existing challenges in multiple sclerosis (MS) studies: 1) Current Atlas-based registration tools require manual vertebra-level labels to function 2) Approaches based on whole spine segmentation for sagittal images lead to sub-optimal axial vertebra instance segmentation due to large slice thicknesses.
This paper proposes a super-resolution-driven solution that enhances the quality of axial images for segmentation to the exact resolution as the inplane resolution of a sagittal slice. The approach involves using sagittal images only for training, avoiding the difficulties associated with aligning sagittal and axial images. 

The proposed approach improves the pre-existing degradation model SMORE based on scan properties to learn a prior that accommodates the distribution shifts between axial and sagittal view images.
main_review: main_review: The authors have addressed most of my comments and have resolved the ambiguity with the Betti matching error metric. Therefore, I would like to stick to my original rating of "weak accept" and finalize it.
strength_weakness: strength_weakness: Due to the large partial volume effects and inter-vertebral movement, we could not register the axial and sagittal image beyond the automatic alignment with an error of up to 30 mm. —-----------> A brief illustration of this challenge needs to be added at least in the appendix.

The breadth of comparative experiments could be better, meaning only two DL methods are compared. The authors have taken much of the allocated space for discussing degradations, squeezing the experiment section. To my knowledge, MIDL has seen manuscripts with extensive experiment sections so far.

The method is a direct application of the Palette with the degradation models inspired by SMORE. As such, the contributions of the work can be explicitly mentioned to improve the readability.

The authors mention the evaluation metric Betti error in most parts of the manuscript. I understand that they used Betti number error from the table captions. The referred paper has the metric Betti matching error, which clearly shows that Betti number error is indecisive where Dice scores are identical, even for a trivial case containing two digits (like MNIST). The evaluation of the efficacy of Betti number error on MNIST dataset with multiple datasets might give a better picture of how reliable this metric is.
questions: questions: There are also some questions in the weaknesses section.

In Section Related work - 
The SMORE zero-shot approach is not suitable in our case due to the lack of rotation invariance and large-scale factors requiring the introduction of globally consistent information. —----> Can the authors rewrite this sentence related to “rotation invariance and large-scale factors” for clarity??

Is there no publicly available dataset for this, for instance, Zenodo spine for MRI?

Ablation studies look good, but because there are so many degradation models, which combinations are clinically relevant?? For example, SMORE one PSF and multi-PSF?
paper_summary: paper_summary: The paper proposes an unpaired inpainting superresolution algorithm that extrapolates missing spine structures in axial and sagittal MRI images. Synthetic training pairs are generated to model the data shift and acquisition errors between sagittal slices and sagittal views of axial images. Overall, the paper presents a novel approach for improving spine MRI segmentation and processing, addressing challenges related to image resolution and acquisition errors.
main_review: main_review: Spine MRIs are often obtained in 2D axial or sagittal slices, which may not fully capture the vertebra structures. Aligning scans for multi-image superresolution is challenging due to volume effects and inter-vertebral movement during acquisition. This study introduces an unpaired inpainting superresolution algorithm that extrapolates missing spine structures. By applying the superresolution model to the z-dimension, it becomes possible to generate images that facilitate automatic segmentation and processing of axial images. The authors have satisfactorily addressed all of my concerns and I have no further comments.
strength_weakness: strength_weakness: - The paper does not provide a detailed analysis of the limitations or potential drawbacks of the proposed unpaired inpainting superresolution algorithm and its impact on the accuracy of the generated synthetic training pairs.

- The evaluation of the proposed method is limited to experimental benchmarks, and there is no comparison or analysis of the algorithm's performance on real clinical data or in comparison to other existing segmentation methods.

- The paper does not provide a comprehensive analysis of the potential impact of the synthetic training pairs and the various degradation functions on the overall segmentation accuracy and robustness of the proposed method.
questions: questions: 1. What are the limitations of the proposed algorithm?

2. How is the algorithm working on real clinical data?

3. What are the potential applications of the algorithm in clinical settings?

4. How does the proposed unpaired inpainting superresolution algorithm generate synthetic training pairs?

5. What are the multiple degradation functions used to model the data shift and acquisition errors?

6. How does diffusion-based superresolution outperform state-of-the-art models in the experimental results?

Comments: comment: The authors have addressed all of my concerns. I have no more comments.
> The paper does not provide a detailed analysis of the limitations or potential drawbacks of the proposed unpaired inpainting superresolution algorithm and its impact on the accuracy of the generated synthetic training pairs. What are the limitations of the proposed algorithm?

The large scaling factor compels the model to incorporate information not present in the original axial image, such as the thickness and hydration level of the intervertebral disk. Since the intervertebral disk can be smaller than a single axial slice, it's unrealistic to expect a perfect match with the real structure. To compensate for this, joint modeling of axial and sagittal images would be necessary, but we have not explored this possibility in this paper. With the segmentation and super-resolution capabilities now available, we can further refine our method. In the revision, we have included the statement: “SR for large scaling factors is strongly ill-posed, and our model is limited to a best guess. The structures between slices, as expected, vary from the real structure.”

> The evaluation of the proposed method is limited to experimental benchmarks, and there is no comparison or analysis of the algorithm's performance on real clinical data or in comparison to other existing segmentation methods. And: How is the algorithm working on real clinical data?

All evaluations are conducted using real clinical data collected from multiple sclerosis (MS) patients treated at clinic “Rechts der Isar” (under approval from the ethics board). Artificial pairs are generated solely for training purposes. Currently, there is no publicly available training dataset for axial segmentation, nor are there any pretrained segmentation models specifically tailored for axial images.

> The paper does not provide a comprehensive analysis of the potential impact of the synthetic training pairs and the various degradation functions on the overall segmentation accuracy and robustness of the proposed method.

The limitation of our analysis comes from the fact that the segmentation for axial images does not exist yet. With the SR we can generate these for the first time. All test metrics we use at least indirectly measure accuracy and robustness. 

Additionally, we ran the SR including segmentation on 360 subjects with two sessions for a follow up study. We found a labeling error in less than <1 % of the cases caused by the model. 

> What are the potential applications of the algorithm in clinical settings?

Thank you for this important question. We feel there are many applications in a clinical context: We are able to conduct single image SR and segmentation on axial images. These are important for enabling segmentation, feature extraction and registration. These are prerequisites for automatic analysis for spine images, for instance in (1) longitudinal progression analysis of patients (2) radiomics and normative value analysis. For instance, the generated segmentations are used to register and compute the location of MS lesion on this cohort for a prospective study. Without this work, a manual annotation of all vertebra heights in all (2 x 360 Images) would have been required to do this study. Generating vertebra height labeling is time consuming and hard to obtain with this low resolution.

> How does the proposed unpaired inpainting super resolution algorithm generate synthetic training pairs?

Based on our proposed degradation scheme, we generate synthetic training pairs by degrading real sagittal images to resemble the appearance of a sagittal view of an axial image.

> What are the multiple degradation functions used to model the data shift and acquisition errors?

A SMORE degradation function is derived from two parameters: the input low resolution and the target high resolution. We varied the input resolution to maintain invariance to the starting resolution. Each sample utilizes only one SMORE filter. 

> How does diffusion-based super resolution outperform state-of-the-art models in the experimental results?

Diffusion-based super resolution's superiority has been demonstrated in various studies, including our experiments detailed in table 1. The assumption is that diffusion is less prone to mode-collapse, better at representing the data distribution, and more responsive to conditioning.
We thank the Reviewer for their feedback and comments.

> A brief illustration of [partial volume effect, inter-vertebral movement and alignment error] challenge needs to be added at least in the appendix.

Thank you for the valuable feedback, we agree that this improves the current manuscript. We have added an illustration (c.f. Fig. 7 in updated manuscript) indicating the mismatch in field of view (FOV) and exemplary misalignment.

> The breadth of comparative experiments could be better, meaning only two DL methods are compared. The authors have taken much of the allocated space for discussing degradations, squeezing the experiment section. 

In order to highlight the main contribution of the paper, we deliberately limited the comparative study of different models because our contribution primarily lies in the degradation function and enabling SR under unpaired constraints.

> The methods compared are only one GAN-based and a CNN, RCAN. The authors should also compare emerging transformer-based models in MRI super-resolution. 

Upon reviewer feedback, we have additionally tested HAT(arXiv:2205.04437) and PTNet(arXiv:2105.13993). Initially, both HAT and PTNet did not converge with the default hyperparameters. Subsequently, we modified HAT to use 4x4 blocks instead of 6x6 and achieved promising results. These have been included in Table1. Notably, training and inference are slower compared to our diffusion model. 

Vertebra dtr$\uparrow$  0.9805
 
Betti $b_0$ er$\downarrow$ 0.098
 
Betti $b_1$ er$\downarrow$ 0.240
 
Axial Dice$\uparrow$ 0.681

> The method is a direct application of the Palette with the degradation models inspired by SMORE. As such, the contributions of the work can be explicitly mentioned to improve readability. (plus, the typo of plain)

Thanks for the hint. We've highlighted this as a final paragraph in the introduction to clarify the adaptation of SMORE's zero-shot learning approach and the use of the conditional diffusion model Palette.

“We change the zero-shot-learning approach of SMORE to an unpaired sagittal **plane** of axial to sagittal super-resolution. We compare existing super-resolution networks **with the conditional diffusion models Palette.**”

> The authors mention the evaluation metric Betti error in most parts of the manuscript. I understand that they used Betti number error from the table captions. […]

Sorry for the ambiguity. We calculate a weak-form of Betti matching error by computing the error per-vertebra. $b_1$ serves as a lower-bound of the true matching error. $b_0$ is always measured correctly by construction.

> [...] the authors are encouraged to try out MRI-Motion-Artifact-Simulation-Tool

Thank you. We will explore this tool in the context of future work.

> The authors measure the segmentation performance of a pre-trained UNet on the different super-resolved axial images. —----> Please elaborate on this sentence

You're correct; the word "different" is inappropriate. We've simplified the sentence to " The axial SR is segmented by a pre-trained network." referring to a preliminary version from Möller et al. (2024).

>  In Section Related work – […] Can the authors rewrite this sentence related to “rotation invariance and large-scale factors” for clarity?

Thank you for the suggestion, we have revised the sentence accordingly (c.f. updated manuscript). 

> Is there no publicly available dataset for this, for instance, Zenodo spine for MRI?

We are not aware of any publicly available dataset for axial data. Li et al. (2021) is the only paper we found, but their data is not accessible. For sagittal spine, Spider and MRSpineSeg Challenge datasets are available, but they exclusively focus on the lumbar spine. We aim to address the lack of annotation/segmentation nets for axial MR Spine. The ability to segment whole sagittal MRI spine has been only released to the public recently, specifically in Graf et al. (2023) and Möller et al. (2024). (citations c.f. paper).

> Ablation studies look good, but because there are so many degradation models, which combinations are clinically relevant?

Some degradation functions are intended to be invariant to changes in input images, while others should be more generalizable to different resolutions. All degradation functions are clinically relevant as they make the model generalize to a specific disturbance.  

> There are high scopes to extend the work as a journal. However, experiments and evaluation metrics can be expanded for further understanding.

We agree and believe that extending the work to other MRI regions where isotropic ground truth exists would be valuable and enhance the results in future studies.
Thank you sincerely for your valuable feedback. We would like to address your concerns by showing you that our **motivation and goals** are multifaceted.

**Improving Resolution:** Our aim is to enhance the resolution of spine images, particularly in non-isotropic scans. This is crucial because obtaining isotropic whole-spine images is often impractical. It's important to note that we're **not seeking to replace sagittal scans**. Both sagittal and axial images offer unique information, e.g. axial images better visualize facet joints, spinal cord lesions, nerve root ganglia and more. By enhancing both, we strive to achieve similar information content to that of an isotropic image. This improvement has multiple medical benefits, especially as the left-right coverage of sagittal scans is always limited: outer parts of the vertebra are never visible. Some large-scale studies like the UKBB lack sagittal images at all. As requested, we've included a new Figure 7 to illustrate axial and sagittal mismatch. 

**Axial Segmentation:** Currently, there's a lack of publicly available data or models for axial spine MRI. While Li et al. (2021) conducted semantic segmentation, their data is not public. Moreover, there's been no attempt at instance label segmentation. Our motivation for employing this approach is to generate segmentations for axial images based on the same principles used for sagittal MR and CT images.

**Feature Extraction:** With our current methodology, we're able to provide semantic and instance segmentation for various downstream tasks. Since the vertebra structure is not sufficiently resolved in axial images, segmentation must be conducted in a super-resolved space. These feature points are invaluable e.g. for biomechanical modeling and point-based registration. We're now able to register axial images to the PAM50 atlas for comparative studies what was not possible before.

In our introduction, we mention MS because the results described here are directly applicable to an upcoming study, where vertebral points of interest are needed to register axial images to an atlas, enabling us to assess the location of white and gray matter lesions along the spinal cord (refer to Fig. 5/6). Manual annotation in axial slices is challenging even for experienced radiologists, error-prone by the limited superior-inferior resolution, and time-consuming.

We hope this clarifies our goals and motivations.
> [....] the comparison should be between a segmentation obtained from a super-resolved image and a sagittal acquisition.

While we agree that a comparison between segmentations of the SR image and the sag. T2w acquisition would be desirable, we cannot provide this comparison, as resampling would lose a lot of detail and we cannot align the sag. and axial SR images.

> The main claim of this article is that sagittal acquisitions cannot be used together with axial acquisitions. […] This claim is not substantiated in this article. 

In discussions with our radiologist, registration experts, and other spine groups worldwide, we assessed the current status of deformable spine registration. Consensus was reached that spine registration remains largely unresolved in most cases without robust supervision such as segmentations or feature points, or significant manual intervention. We utilized various packages including Nipy, AirLab, ANTs, and SimpleITK for evaluation. Assessment was based on spinal cord Dice scores and visual alignment of vertebra corners. Unfortunately, registration either maintained the initial Dice score or deteriorated it. Despite attempting masking strategies for the spinal cord and incorporating spinal cord segmentation, alignment could not be improved due to significant discrepancies in pixel volumes. We also consulted multiple researchers regarding software capable of registering these images, but satisfactory results could not be achieved.

> [...] the main novelty of the presented work is the degradation process. […]

The main innovation lies in the degradation process and the methodology for training a SR model within our specified constraints. Joint unpaired and inpainting SR represent specialized cases that have received less attention in the medical super-resolution community. Additionally, our paper delves into the downstream tasks made possible by our constrained SR approach.

> […] Therefore, the comparisons are actually focused towards showing performance gains due to Palette. […] 

We agree. Table 1 compares SR models to validate our choice of model. The performance gains are investigated in the experiments of Table 2. 

>[...] degradation, synthesis as well as segmentation in 2D or 3D?

Apologies for the misunderstanding. We have clarified this in our manuscript. According to the physical principles of image acquisition, where we focused solely on typical 2D sequences commonly available for spine imaging, the degradation process had to be conducted in 2D. Meanwhile, the segmentation is 3D.