Automated diagnostic of cervical spondylosis on multimodal medical images with a multi-task deep learning model

Domain: neuroscience
Published: 2026-02-05
Editorial outcome: major_revision

Abstract
Cervical spondylosis is one of the most common degenerative diseases, seriously affecting life quality. Unlike diseases with explicit lesions like cancer, hydroncus, or fracture, the degeneration of the cervical spine cannot be explicitly detected from the appearance of medical images, requiring extensive experience of doctors to interpret subtle clues. However, the extremely high incidence of cervical spondylosis coincides with a serious shortage of experienced doctors and uneven distribution of medical resources, hindering early diagnosis. We propose a cascade-ensemble deep learning framework for cervical spondylosis diagnosis. The framework integrates vertebral body detection and degenerative diagnosis through a cascading architecture, and jointly trains an ensemble of degenerative indicators in a multi-task learning manner. We demonstrate that deep learning models are more sensitive to distance and position based indicators than angle based ones. In intervertebral stenosis analysis, our method achieves comparable performance to senior radiologists and clinicians, with much faster diagnostic speed.

Introduction
Cervical spondylosis is a seriously harmful disease occurring in any population, seriously affecting the health and quality of our lives. The diagnosis and treatment of cervical spondylosis usually require a combination of imaging and clinical symptoms. Cervical spondylosis is a degenerative disease that causes clinical manifestations such as neck pain, stiffness, headaches, numbness, tingling, and weakness in the arms or legs due to cervical degeneration in different parts. In the World Health Organization (WHO)’s “Global Top Ten Persistent Diseases", cervical spondylosis ranks second. Among the global population, the number of people with cervical spondylosis is as high as 900 million. The prevalence of cervical spondylosis was 13.76%1 among 3900 selected participants, and this ratio differed significantly among the urban, suburban, and rural populations (13.07%, 15.97%, and 12.25%, respectively). With changes in lifestyle and working environments, cervical spondylosis is no longer a common condition among middle-aged and elderly people. The East Asia and Andean Latin America regions had the highest and lowest age-standardized incidence rates, with 1029.0 (910.5 to 1166.1) and 624.0 (550.3 to 708.3) per 100,000 population in 20172.

Compared to diseases with explicit lesions, such as cancer, hydroncus, or fracture, cervical spondylosis is a chronic, primarily non-traumatic and progressive condition without explicit lesions, which are usually diagnosed with the combination of appearance and professional metric measurement in clinical practice. According to the affected structures and clinical manifestations, cervical spondylosis could be caused by different types of degeneration, such as the degeneration of the cervical vertebrae curvature, vertebral instability, intervertebral disks and the intervertebral joints in the cervical spine, which compress or stimulate the adjacent spinal cord, nerve roots, and vertebral arteries, leading to the symptoms of arm pain, numbness, neck pain, weakness, incontinence, and impairment of gait (see Supplementary Table 1 in the Supplementary Information).

In particular, the pathological forms of cervical degeneration are diverse and vary greatly; correspondingly, the characterizations of different degradation phenomena on medical imaging also vary. Similarly, the therapeutic schedule of different cervical spondylosis types varies greatly, and accurate preoperative imaging is vital to determine the cause of compression as soon as possible to determine an appropriate surgical plan.

Common examinations for cervical spondylosis include X-rays, computed tomography (CT), and magnetic resonance imaging (MRI). CT is suitable for displaying changes in the spinal structure, but it exposes the body to relatively high dose of radiation. MRI shows the fine details of soft tissues, but it is time-consuming and expensive; in addition, patients with metal in their body may have contraindications. Compared with CT and MRI, X-ray radiation is cheaper, more convenient, and faster. More importantly, based on clinical experience, orthopedic experts have artificially set sagittal plane parameters for cervical spine X-rays, which have become key factors in determining surgical options for cervical spondylosis and have been widely used clinically. For example, if the X-ray shows cervical kyphosis, it is suggested to choose anterior surgery to improve the curvature; if it shows cervical spinal canal stenosis, it prompts the surgeon to choose posterior spinal canal enlargement surgery to relieve spinal cord compression; if it shows cervical segmental instability, it indicates the need for metal internal fixation to enhance stability.

In general, failure to correctly identify the sagittal factors may lead to wrong-procedure surgery and poor prognosis. However, an accurate diagnostic of cervical spondylosis requires extensive clinical experience. On the one hand, the distribution of medical resources in most countries is uneven, and senior doctors are concentrated in large cities at the nation’s top-class hospitals. While the experience of physicians at primary hospitals is limited, the diagnosis and treatment efficiency for cervical spondylosis may not be guaranteed. On the other hand, the workload of senior doctors has long been oversaturated, and the resources for on-site guidance to the primary hospitals are limited. Thus, how to transmit the experience accumulated over the years at tertiary and top-class hospitals in treating cervical spondylosis to primary hospitals is urgently needed, which will be beneficial to the majority of patients. In addition to the on-site guidance, a deep learning model trained with expert doctors’ experience, used as supervision in training, could be a solution to improve the diagnostic accuracy in primary hospitals, meanwhile, significantly saving diagnostic time for all hospitals.

Existing deep learning based studies related to the assessment of cervical degeneration can be broadly categorized into the localization-based and semantic segmentation-based approaches for degeneration assessment. Among them, some localization-based works are proposed with X-ray images to determine the presence of certain types of degeneration, such as the ossification of the posterior longitudinal ligament3 or spinal stenosis4, or the degree of spinal cord compression5. However, there are multiple factors that lead to the degeneration of the cervical spine, and the assessment of the overall situation is still mostly based on manual measurements6,7. Moreover, the methods8,9 of localizing body structures, such as cervical vertebrae, mainly focus on the center of the vertebral body, but ignore the edges and corner points and lack further assessment. Alternatively, some semantic segmentation-based studies10,11 have performed meticulous semantic segmentation of cervical spine structures, such as bones or disks, which extremely rely on the large scale of annotated data, and the edges of vertebral segments are usually not clear enough to meet the precision requirements from the doctors. In recent years, the methods of feature point detection have become popular for medical diagnostic tasks such as head12, hand13, lower limb14 and hipbone15. However, in contrast to the landmark detection used in the scenarios where bones are susceptible to differentiation, the cervical vertebrae are highly similar to each other, and are likely to result in the projection artifacts and structural overlap. In general, previous deep learning methods have made attempts on cervical analysis from some particular aspects, such as vertebral detection, spinal stenosis, intervertebral disc degeneration, those models for separate degenerative factors still could not meet the practical requirements of doctors. One intuitive idea could be combining all those models in a system. However, considering different models are designed in different architectures and trained with different data, directly combining previous models is difficult to achieve.

In this work, we propose a multi-modal collaboration cervical spondylosis diagnosis framework trained with multi-task learning, oriented to the clinical requirements of less annotation workload for doctors, and fine-grained measurements of multiple types of degeneration. According to the patient's symptoms (see Supplementary Table 1 in the Supplementary Information) and the expert knowledge, the cervical vertebrae curvature, vertebral instability, spinal stenosis, and intervertebral stenosis are selected for the diagnosis of cervical spondylosis. Considering different degenerative factors are applicable to different medical imaging modalities, each degenerative factor is diagnosed with the most suitable modality in the proposed framework. The proposed architecture is designed with cascade and ensemble networks, where the location and diagnosis modules are implemented with a cascade architecture, and the diagnosis of different degenerative diseases is implemented as an ensemble network. Note that all the models were trained with the imaging data of the same patients, where each patient contains X-ray and MRI data. The multi-modal diagnostic results collaborate complementarily to support the doctors’ needs, reflecting real-world clinical practice where doctors integrate findings from different imaging modalities to address distinct aspects of the diagnostic process. The proposed method is capable of producing a comprehensive report detailing potential cervical degeneration and their respective positions, fulfilling the pragmatic requirements of medical practitioners more effectively.

In conclusion, we designed our multimodal diagnostic framework of cervical spondylosis from the clinical requests of doctors. Typical cervical spondylosis indicators, such as Cobb angle, SVA value, intervertebral foramen stenosis, and spinal stenosis, are required to be explicitly provided for diagnosis. Different cervical spondylosis indicators are best suited to be acquired through different data modalities. We design our framework with multimodal collaboration instead of fusion to one overall result. The key contribution of our work to doctors lies not only in providing results from a multimodal perspective, but also in offering the potential for mutual confirmation among different indicators through a multitask learning approach.

Results
Patient characteristics and dataset composition
This retrospective study collected cervical spine X-ray and MRI scans of 1500 patients (mean age ± standard deviation: 54 years ± 10.28, men: 909, 60.60%, women: 591, 39.40%) with cervical spondylosis at the Third Hospital of Peking University from January 2016 to December 2018. During the preparation of data, some patients were excluded due to inappropriate labeling, such as missing vertebral bodies or spinous processes, resulting in a total of 1151 patients being used. Data collection was pre-approved by the Institutional Review Board and excluded minors as well as patients with instrumentation or other medical conditions (e.g., spinal tumors, infections, trauma, or scoliosis). They were annotated by three radiologists and randomly divided into a training set (693), a validation set (228) and a testing set (230). For the vast majority of cases, the three annotators’ results are largely consistent. In cases of disagreement, the annotations from a more authoritative expert shall prevail. All data were manually labeled by radiologists and validated against each other. The labeling information was all by the four-point method of vertebrae in lateral and flexion-extension positions on cervical spine radiographs. Each radiologist was blinded to patient demographics and clinical history. The dataset contained one cervical spine radiograph of each patient in three positions: neutral, flexion, and extension in the lateral position.

Vertebral detection on X-ray Images
In our case, the vertebral bodies are represented as quadrangles, it’s also equivalent to detecting the four vertices of the quadrangles. The detection performance is reported as the error distribution between the predicted vertices to the annotated vertices in three views, including flexion, neutral and extension in Fig. 1a–c. In most cases, from C3 to C7, the mean error is below 1 mm. In particular, the upper margin of the C2 and the down margin of C7 have larger variance, the mean of the former is slightly over 1 mm, and the mean of the latter is close to 1 mm. Also, several outliers with larger errors are observed; correspondingly, C2 and C7 contain more outliers than the others.

Cervical Vertebrae Curvature (CVC) analysis on X-ray images
The C2-7 Cobb Angle (CA) analysis method was developed according to the policy in 6,7,16,17, i.e., the angle between the lower end plate of C2 and the lower end plate of C7, where the vertebral bodies are detected with the model in subsection 2.1. The results of the Cobb angle are illustrated in Fig. 2, where the sub-figures Fig. 2a–c show the numerical analysis of Cobb angle estimation. The mean error of our method is 2.42°, with the median of 1.84°, and over 80% of samples were predicted with an error under 4°, the outliers over 8° were less than 1.5% (see sub-figure Fig. 2a), and the first and third quadrilles were close to 1° and 3° (see sub-figures Fig. 2b). The sub-figure Fig. 2c shows the bias distribution between our predicted results and the gold standard (of Cobb angle) with Bland-Altman analysis, and the 95% limits of agreement (±1.96 SD) were [−6.49°, 6.31°]. In the Bland-Altman analysis, if most of the points fall within the 95% consistency limit (i.e., within the two lines of mean ± 1.96 * standard deviation), and their maximum difference is clinically acceptable, then ‘the compared two methods’ can be considered to have good consistency and can be replaced with each other. In this work, our model and radiologist could be considered as having ‘good consistency’, since most points fall into the range between the 95% limits of agreement.

Sub-figure Fig. 2d illustrates the accuracy of clinical recommendations according to 18. The Cobb angle is a crucial metric for assessing cervical spine curvature. It can reflect the lordosis (forward curvature) and kyphosis (backward curvature) of the cervical spine. Research conducted by Suda et al.18 has demonstrated that, through a multivariate logistic regression model, when the local kyphosis angle exceeds 13°, meaning the Cobb angle is less than −13°, successful surgery becomes challenging. Therefore, addressing the kyphosis issue should be prioritized. Conversely, when the local kyphosis angle is less than 13°, indicating a Cobb angle higher than −13°, the likelihood of surgical success is higher. Therefore, we have established a clinical threshold of −13° for the Cobb angle as a guideline. If the Cobb angle is less than −13°, surgery is not recommended, while if it is greater than 13°, surgical intervention may be considered. The accuracy distribution shows the proposed method could give a 100% accurate suggestion of cervical kyphosis, for the patients with Cobb Angle (gold standard) under −20° or over −10°. For the patient with a Cobb angle in [−20°, −10°], the accuracy is about 75.4%.

The C2-7 SVA value analysis method was developed according to the policy in 19, i.e., the distance from the center vertical line of C2 to the posterior superior angle of C7. The results of the SVA value are illustrated in Fig. 3, where the sub-figures Fig. 3a–c show the numerical analysis of SVA estimation. The mean error of our method is 0.65 mm, with the median of 0.50 mm, and over 80% samples were predicted with the error under 1 mm, the outliers over 2.5 mm were less than 4% (see sub-figure Fig. 3a), and the first and third quadrilles were close to 0.3 mm and 0.8 mm (see sub-figures Fig. 3b). The sub-figure Fig. 3c shows the bias distribution between our predicted results and the gold standard (of SVA value) with Bland-Altman analysis, and the 95% limits of agreement (±1.96 SD) was [−1.58 mm, 1.78 mm].

SVA (Sagittal Vertical Axis) is a crucial factor for measuring cervical spine curvature. According to recent research 20, C2–C7 SVA more than 40 mm can be considered indicative of Cervical Deformity (CD). Therefore, we set the SVA threshold at 40 mm as the clinical assessment criterion. If SVA is more than 40 mm, it is generally considered a cervical curvature imbalance, which is also considered a kyphosis deformity; thus, surgical intervention may be implemented to correct cervical spine deformity. Also, according to decades of experience of Peking University Third Hospital, SVA less than 15 mm can also be the clinical assessment criterion. The accuracy distribution shows the proposed method could give a 100% accurate suggestion of cervical kyphosis for patients with SVA value (gold standard) under 10 mm or over 20 mm. For the patient with an SVA value in [10 mm, 20 mm], the accuracy is about 90.3%.

Vertebral instability analysis on X-ray images
The spondylolisthesis angle was estimated as the absolute difference between two views of the angle of extension of the posterior edge of the adjacent vertebral body. The results of the spondylolisthesis angle were illustrated in Fig. 4a; the errors are distributed between different vertebral segments, and there was a trend with larger errors from C2-C3 to C6-C7. The mean error of C2-C3/C6-C7 is from 2° to 3.6°.

The spondylolisthesis distance is estimated as the absolute difference between the two types of distance: 1) the distance between the extension line of the posterior edge of the upper vertebra and 2) the parallel line passing through the posterior superior corner of the lower vertebra. The results of the spondylolisthesis angle were illustrated in Fig. 4b, the errors are distributed between different vertebral segments, among all the segments, the segment C5-C6 have the smallest mean error about 0.4 mm, while the segment C2-C3 have the largest mean error about 0.6 mm.

In terms of vertebral instability, we evaluate it based on the spondylolisthesis angle and distance. According to the guideline in 21, the instability of the cervical vertebral segments was defined when there was more than 3.5 mm anterior or posterior translation or more than a 11° difference in spondylolisthesis angle. Therefore, we conducted the vertebral instability evaluation with the classification criteria of 3.5 mm in spondylolisthesis distance and 11° in spondylolisthesis angle. Meaning that when the spondylolisthesis distance exceeds 3.5 mm or the spondylolisthesis angle is more than 11°, it is considered as vertebral instability, and anterior cervical discectomy and fusion treatment may be required.

The classification results are illustrated in Fig. 4c with a Radar chart, which is also more convenient for comparison. It can be observed that the classification accuracy with spondylolisthesis distance on each segment outperforms the accuracy with spondylolisthesis angle; the overall accuracy of spondylolisthesis angle/distance is 80.5/99.4%.

Spinal stenosis analysis on X-ray images
The Sagittal Diameter of Vertebral Body (SDVB) was estimated as the distance from the midpoint of the anterior edge of the vertebral body to the midpoint of the posterior edge. The results of SDVB are illustrated in Fig. 5a, the mean errors estimated by our model are as follows: 1.67 mm for C3, 0.94 mm for C4, 0.77 mm for C5, 0.45 mm for C6, and 0.70 mm for C7.

The Sagittal Diameter of Vertebral Canal (SDVC) was estimated as the shortest distance from the midpoint of the posterior edge of the vertebral body to the lamina. The results of SDVC are illustrated in Fig. 5a, the mean errors estimated by our model are as follows: 2.46 mm for C3, 2.43 mm for C4, 2.14 mm for C5, 1.71 mm for C6, and 2.60 mm for C7.

The Pavlov value was estimated as the ratio of SDVC/SDVB. The results of Pavlov ratio are illustrated in Fig. 5c, the mean errors estimated by our model are as follows: 0.09 for C3, 0.11 for C4, 0.09 for C5, 0.08 for C6, and 0.10 for C7.

Pavlov value is significant in assessing developmental cervical canal stenosis. When the Pavlov value of three consecutive vertebral segments is less than 0.75, developmental cervical spinal stenosis is diagnosed. According to the guideline in 22, Wang et al. attempted to categorize the patients into two groups, Developmental Canal Stenosis (DCS) and Non-Developmental Canal Stenosis (NDCS), based on the guideline of whether the Pavlov value is less than 0.75 or not. Thus, crossing 0.75 of the Pavlov value is a crucial clinical determinant; patients consisting of three consecutive vertebral segments with a Pavlov value less than 0.75 are considered as the indicative of developmental canal stenosis, and posterior cervical surgery may be considered.

Intervertebral degeneration analysis on MR images
The results of our model in analyzing the intervertebral disc degeneration (i.e., normal or bulge vs. protrusion or extrusion) are illustrated in Fig. 6a, resulting in the accuracy of 76.3% and the Area Under the Curve (AUC) of 0.844. Compared to the ROC curve of our model, all the results of doctors were under the curve of our model, demonstrating that our model could achieve better sensitivity or specificity in a similar scenario. When comparing to the doctors in more detailed Radar chart, our model outperformed the clinical doctors and radiologists in the evaluation measurement of accuracy and F1 score. The clinical doctor outperformed the radiologist and our model in specificity.

The results of spinal stenosis analysis are shown in Fig. 6b, with the AUC of 0.925, and accuracy of 87.6%, which is better than both clinical doctors and radiologists. When comparing to the doctors in a more detailed Radar chart, our model outperformed the clinical doctors and radiologists in the evaluation measurement of accuracy, sensitivity and F1 score. The clinical doctor outperformed the radiologist and our model in precision.

The results of intervertebal foramen stenosis analysis are illustrated in Fig. 6c, d, and the AUC of left and right intervertebal foramen stenosis diagnostics were 0.849 and 0.845, which were very closed to the AUC of intervertebral disc degeneration diagnostic. Comparing to the doctors in ROC curve, the points of clinical doctor were above the curve of our model, and the points of both junior and senior radiologists were under the curve of our model. When comparing to the doctors in the Radar chart, it could be seen that our model performed better in accuracy and F1 score, and the clinical doctor diagnosed better in precision and specificity. According to our statistic analysis, intervertebral degeneration usually takes about 3 minus for the radiologist, but it takes less than 2 s for our model.

The patient level diagnostic analysis
The patient-level results are illustrated in Fig. 7 as a radio chart, and also the visualization of each degenerative factor. Note that these results are calculated in patients, instead of being calculated in images in the above results, where the degeneration correctly detected in any sequence could be regarded as a correct diagnosis. Each axis in the radar chart represent the diagnostic accuracy of the corresponding degenerative factors. The overall average accuracy was 87.6%, and the diagnostic accuracy of other diseases was in the range of [75.1%, 98.5%]. Also, a real diagnosis case of cervical spondylosis is illustrated in Fig. 8, where all the diseases diagnosed as positive are illustrated. In the case of Fig. 8, the cervical spine degeneration was detected as spinal stenosis in C4-5, and unstable in C5-6, cervical disc herniation in C3-C6, and spinal canal stenosis in C4-5. To some extent, C3-C6 Cervical disc herniation is a direct cause of Spinal canal stenosis. As the degenerative changes in the cervical spine of the patient progress, multiple disks at C3-C6 gradually undergo herniation, reducing the effective space within the spinal canal, thereby forming cervical spinal stenosis and ultimately leading to compression of the cervical spinal cord or nerve roots. At the same time, Spinal canal stenosis is also a manifestation of the compression of the spinal cord by C3-C6 Cervical disc herniation, and the two conditions can corroborate each other.

The external validation
We also conducted the external validation with the patients in Beijing Haidian Hospital, the results are illustrated in Fig. 9. In general, the overall average accuracy is 85.8%, which is very close to that of Peking University Third Hospital (87.6%). Compared to the validation within Peking University Third Hospital, the performance of both our model and the clinical doctors were somewhat lower than that in Fig. 6. In particular, the AUC of intervertebral disc degeneration diagnostic is 0.808, and the AUC on left and right foramen stenosis were 0.707 and 0.705 in Fig. 9c, d, where the ‘points’ of the clinical doctor were near ROC curve of our model. The clinical doctor slightly outperforms our model.

Methods
This study was approved by the Peking University Third Hospital Medical Science Research Ethics Committee, which waived the requirement for informed consent. This waiver was granted as our study was a retrospective analysis of pre-existing imaging data. The research is strictly non-commercial and uses fully anonymized data with all personal identifiers permanently removed. The study was performed in accordance with national and international guidelines.

The proposed method is designed in cascade and ensemble architecture, where detection module and diagnostic module are cascaded (see Fig. 7). Different degenerative indicators are integrated in an ensemble module, which are contributed as different losses in deep learning model training. This section introduces the proposed framework in two parts, detection and degeneration diagnosis in multi-task learning.

Detection
The input to the vertebral body detection model is the image I∈RC×W×H, where C, W, and H denote the number of channels, width, and height of the input image. The output is the locations of K keypoints, K is 24 (i.e., the four corner points of vertebral bodies from C2 to C7), denoted as p k K k=1 ,p k= x k,y k . Since the input radiographic image is a grayscale image, the number of channels C is 1.

Previous keypoint detection models37 usually follow a similar pipeline to extract the image features for the k-th keypoint first, and then predict a heatmap H b k 2Rw×h for the k-th keypoint, where w×h denotes the size of the output heatmap. Then, obtain the pixel position p k of the k-th keypoint by argmax, and finally group the keypoints of different instances.

The ground truth pixel position is usually converted into a heatmap, denoted as H k, which is generated by a 2D Gaussian function with a standard deviation of 1. The loss for heatmap supervision is computed using the Mean Squared Error (MSE) between the ground-truth heatmap H and the predicted one H b . The calculation is performed as follows:

MSE(H,Hb) = 1/N * ΣN i=1 (h i - hb i)^2 (1)

L = L heatmap = MSE(H,Hb) (2)

Heatmaps are typically generated at lower resolutions through down-sampling to balance computational efficiency and receptive field coverage, leading to localization errors. Particularly for medical images, where most targets are relatively small in images, minimizing localization errors becomes critical. Our keypoint detection module considers to generate the heatmap on the highest resolution feature map, inspired by the HigherHRNet38, which is built on HRNet39,40. The high-resolution output obtained by deconvolution upsampling is added to form a high-resolution feature pyramid learning multi-scale perceptual representation. Heatmap loss L heatmap contain multi-resolution supervision and can be expressed as:

L0 heatmap = ΣN n=1 MSE(H nk,Hb nk) (3)

Where N is the number of layers of the feature pyramid, while H nk and Hb nk denote the n-th layer of the heatmap at the k-th keypoint.

Degeneration diagnosis in multi-task learning
In this section, we propose a Multi-task Learning Mechanism (MLM) module, which combines the metric measurement task with the keypoint detection task in an attempt to improve the overall performance without adding additional annotations. In our multi-task learning framework, we employ the Differentiable Spatial to Numerical Transform (DSNT)41 to convert heatmaps into coordinate representations through a differentiable spatial expectation operation, enabling end-to-end backpropagation of errors from multiple clinical indicators to the keypoints detection module. Given the predicted heatmap Hb 2Rh×w, we apply the softmax function for normalization to ensure the results conform to a probability distribution:

Zb i,j = exp(Hb i,j) / Σh k=1 Σw l=1 exp(Hb k,l) (4)

We then construct two h×w matrices X and Y, where X ij = 2j - (w/w+1) and Y ij = 2i - (h/h+1). The matrices X and Y encode normalized coordinate values, where each element contains the x-(for X) or y-coordinate (for Y) in the range of (-1, 1). The predicted coordinates (xb, yb) are derived through the Frobenius inner product between the normalized heatmap Zb and coordinate matrices X, Y.

(xb, yb) = DSNT(Zb) = [⟨Zb, X⟩ F, ⟨Zb, Y⟩ F] (5)

where ⟨⋅,⋅⟩ F denotes the Frobenius inner product, which is equivalent to the scalar dot product of vectorized matrices. Essentially, this operation computes a weighted average of the coordinate grid with respect to the heatmap’s probability distribution, yielding the mathematical expectation coordinates under this distribution. Since both the softmax and expectation operations are differentiable, gradients can be backpropagated seamlessly.

Although DSNT can replace the heatmap output, in order to preserve the multi-resolution heatmap of HigherHRNet, we use the loss for the DSNT-based metric measurement task and loss L0 for the multi-resolution heatmap keypoint detection task, represented as:

L Ours = L multi-task + L0 heatmap = ΣM m=1 λ m MSE(Qm, Q b m) + ΣN n=1 MSE(H nk, Hb nk) (6)

Q = [QCA, QSD, QSA, QSDVB, QSDVC] (7)

Among them, Qm is taken from the coordinates P0 obtained by DSNT mapping, where λ m denotes the parameter of the m-th metric measurement operation, and M denotes the number of metric measurement tasks. Qm denotes the result of the m-th metric measurement with ground truth coordinates. It is explained below how to obtain the metric measurement Qm from the coordinate matrix P0:

P0 = [x1, y1, x2, y2, x3, y3, x4, y4; ...; x21, y21, x22, y22, x23, y23, x24, y24] 6×8 (8)

Points p 1 to 4 represent the antero-superior, antero-inferior, postero-superior, and postero-inferior corner points of the C2 vertebrae, as well as soon.

Assessing cervical curvature relies on two indicators, the C2-7 Cobb angle and the C2-7 SVA. In particular, the angle points of the lower edges of the C2 and C7 vertebrae are used to calculate the Cobb angle. The following values in the corresponding coordinate matrix P0:

p 2x = P0[0,2], p 2y = P0[0,3], p 4x = P0[0,6], p 4y = P0[0,7] (9)

p 22x = P0[5,2], p 22y = P0[5,3], p 24x = P0[5,6], p 24y = P0[5,7] (10)

The detailed formula of QCA is:

k1 = (p 2y - p 4y) / (p 2x - p 4x + δ), k2 = (p 22y - p 24y) / (p 22x - p 24x + δ), δ=1e-9 (11)

QCA = arctan((k1 - k2) / (1 - k1 * k2)) * 180 / π (12)

Similarly, the SVA calculation is dependent on the coordination of the four corners of C2, as well as the postero-superior corner p 23 of C7:

p 1x = P0[0,0], p 1y = P0[0,1], p 2x = P0[0,2], p 2y = P0[0,3] (13)

p 3x = P0[0,4], p 3y = P0[0,5], p 4x = P0[0,6], p 4y = P0[0,7] (14)

p 23x = P0[5,4] (15)

p0 is the coordinates of the diagonal intersection of the C2 vertebrae, and the details of the calculation of the QSVA are as follows:

p0 x = (p 4x * p 2y - p 1y * p 3x) / (p 1x * p 2y - p 1y * p 2x) (16)

QSVA = |p0 x - p 23x| (17)

Vertebral instability can be assessed using the spondylolisthesis angle and spondylolisthesis distance. Both calculations depend on the coordinates of the posterior corners of the adjoining vertebrae, as in the example of C5-6:

p 15x = P0[2,4], p 15y = P0[2,5], p 16x = P0[2,6], p 16y = P0[2,7] (18)

p 19x = P0[3,4], p 19y = P0[3,5], p 20x = P0[3,6], p 20y = P0[3,7] (19)

The value QSD of the spondylolisthesis distance is given by:

A = p 16y - p 15y, B = p 15x - p 16x, C = (p 15y - p 16y) * p 15x - (p 16x - p 15x) * p 15y (20)

QSD = |A * p 20x + B * p 20y + C| / sqrt(A^2 + B^2) (21)

The value QSA of spondylolisthesis angle is given by:

k 1 = (p 15y - p 10y) / (p 15x - p 18x + δ), k 2 = (p 19y - p 20y) / (p 19x - p 20x + δ), δ=1e-9 (22)

QSA = arctan((k 1 - k 2) / (1 - k 1 * k 2)) * 180 / π (23)

The assessment of spinal stenosis relies on the Pavlov value calculated as:

Pavlov = QSDVC / QSDVB (24)

SDVB is the length of the line segment joining the anterior and posterior mid-points of the vertebra, using the C3 vertebra as an example:

p 5x = P0[1,0], p 5y = P0[1,1], p 6x = P0[1,2], p 6y = P0[1,3] (25)

p 7x = P0[1,4], p 7y = P0[1,5], p 8x = P0[1,6], p 8y = P0[1,7] (26)

The QSDVB of C3 was calculated as:

p anterior x = (p 5x + p 6x) / 2, p anterior y = (p 5y + p 6y) / 2 (27)

p posterior x = (p 7x + p 8x) / 2, p posterior y = (p 7y + p 8y) / 2 (28)

QSDVB = distance(p anterior, p posterior) = sqrt((p anterior x - p posterior x)^2 + (p anterior y - p posterior y)^2) (29)

QSDVC is the shortest distance from the posterior midpoint of the vertebra to the anterior cervical spinous process M, and the matrix M is obtained from the segmentation model, which in the example of C3 can be expressed as:

QSDVC = min(distance(p posterior, p2M)) (30)

All above metrics Q=[QCA, QSD, QSA, QSDVB, QSDVC] are jointly participated in multi-task training and optimization for X-rays. Our multi-task framework is implemented as an extension of the Faster R-CNN model42. In the Faster R-CNN model, two tasks of classification and localization are jointly learned as two branches after several shared CNN layers. In our implementation (see Supplementary Fig. 1 in the Supplementary Information), each metric in Q is implemented as a branch as follows:

L multi-task = ΣM m=1 λ m MSE(Qm, Q b m) (31)

The implementation for processing X-ray images is divided into two stages. The first detection stage detects the coordinates of cervical vertebral keypoints, while the second prediction stage predicts multiple degenerative indicators. Our detection module extracts the feature map from the input images and then generates the heatmap on the highest resolution feature map, inspired by the HRNet39,40 and HigherHRNet38. The detection module is a parallel architecture that consists of several branches, where one branch focuses on high-resolution features, while others progressively downsample the input to create low-resolution representations (see Supplementary Fig. 1 in the Supplementary Information). The high-resolution branch consists of several residual blocks that utilize 3×3 convolutions, batch normalization, and ReLU activation. Alongside the high-resolution branch, there are low-resolution branches at different scales (1/2, 1/4 the original resolution), and the downsampling is implemented by the strided convolutions. The architecture also employs a fusion mechanism to combine high-resolution and low-resolution feature maps, using upsampling and convolution operations.

After obtaining the feature map through the above backbone network, we use a deconvolution upsampling operation and several residual convolutional layers to obtain the final high-resolution heatmaps. Then we use the Differentiable Spatial to Numerical Transform (DSNT)41 to get the mapped pixel coordinates from the normalized heatmap prediction, and finally group the keypoints of different instances. Finally, the predicted values of various degenerative indicators are calculated from the linear layers based on the coordinates obtained by the DSNT mapping.

The diagnosis of intervertebral disc degeneration on MRI could also be implemented as a branch after several shared CNN layers. However, considering the visual differences between X-ray and MRI images, we separately train a different CNN model for MRI images (see Supplementary Fig. 2 in the Supplementary Information).

Evaluation metrics
In order to verify the capability of our model in diagnosing cervical spondylosis diseases, we evaluate our method from two aspects: measurement error and recognition performance. The measurement error consists of the Mean Absolute Error (MAE) of angle and distance, which could be calculated as the MAE angle and distance between the predicted results of our model and the golden standard annotated by the expert. The performance metrics are evaluated through accuracy, precision, sensitivity, specificity and F1 score.

Implementation details
The width of the original X-ray images ranges from 908 to 2973 pixels, and the height ranges from 1125 to 3032 pixels. With reference to 43,44, we resized all images uniformly to 512×512 pixels and performed normalization operations to standardize input distributions. Regarding the hyperparameter configuration for model training, we set the batch size to 32. The model was initialized with the pre-trained HigherHRNet38 weights provided by MMPose45, and trained for 300 epochs using the Adam optimizer with a fixed learning rate of 1.5×10−3.

Statistics and reproducibility
This retrospective study was conducted on a cohort of 1151 patients. The dataset was randomly partitioned into training (n=693), validation (n=228), and testing (n=230) sets. No statistical method was used to predetermine the sample size, and no data were excluded following initial eligibility screening. The reproducibility of the analysis is facilitated by the publicly archived code.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The manuscript presents a new deep learning model to automatically assess cervical spondylosis. I enjoyed reading the\nmanuscript but I have a few concerns.\nIntroduction\nAt the end of the Introduction, there is the sentence that describes your model as multi-modal, multi-task, and ensemble. I am\nnot sure if you can use these terms. Multi-modal means that you use different data modalities (images, clinical data, omics...)\nand combine them to improve your prediction and have a full picture of the patient's health status. In your case, you have two\nimage modalities but you use them independently (model for radiographs and model for MRI). Am I missing something?\nCould you clarify? It makes sense to use two image modalities but you are not combining them. In your case, the two\nmodalities are treated separately so there are two stand-alone models.\nRegarding multi-task: why is the model multi-task? You predict coordinates and from your predictions, you compute some\nmeasures. My understanding of multi-task is that you have multiple tasks (i.e. localization + classification) and you optimize\nboth to improve the prediction of the main task. Could you please clarify?\nRegarding ensemble: similar comment. Your model predicts different outcomes (different parameters) while in ensemble\nlearning you have many learners that predict the same outcome and the very final outcome is predicted by\naveraging/voting/.... the predictions by the learners.\nThe general comment is to try to clarify/explain better all the concepts you mentioned (multimodal/multitask/ensemble).\nResults\nIn the first paragraph, it is written '1500 patients'. Then, In the paragraph after there is a reference to 1151 cases. I am\nmissing the consistency of the numbers. Please clarify. Also, you only mentioned radiographs and not MRI data.\nVertebral detection X-ray images\nHow is the error computed? Is it the Euclidean distance?\nPage 8 C2-7 SVA paragraph\nPlease check the references to the figures in all Results paragraphs. Some of them are wrong.\nIntervertebral Degeneration Analysis on MR images\nI do not get how you can say that your model outperformed the clinical assessment as the ground truth to train the model was\ngenerated from clinicians. How was the ground truth computed? Average of annotations? Voting? Consensus? Please\nclarify how the ground truth was generated.\nIntervertebral foramen stenosis\nIn the last sentence, you refer to a statistical analysis to evaluate intervertebral disc assessment. I did not find a reference to\nthis analysis in the Methods and I do not understand what kind of analysis it is. There is also a typo after the '3'.\nThe Patient level Diagnostic Analysis\nCheck again the Figures references.\nDiscussion\nI am not sure that you can compare your work with the works you mentioned in the discussion because they are computing\nthe Cobb angle on the frontal plane for the full spine while you are just using the cervical region. In your case, it could be\neasier to get better results because your Cobb is always calculating between C2 and C7 while the Cobb angle is the\nmaximum angle computed on the frontal plane.\nMethods\n- Since C2 is a vertebra that is quite different from the other vertebrae (it does not have a clear upper endplate), how did you\ndefine the upper endplate corners?\n- Specify what is L and how it is computed.\ngroup\n- Line 20 very last line: I do not get the sentence. What do you mean by 'Particularly for the medical images,.....'\nDegeneration Diagnosis in Multi-task Learning\nAs outlined before, I am not sure that you can talk about multi task approach. Please explain.\n- Page 24 last sentence before Evaluation metrics paragraph: The sentence seems out of context. You described all the\ngeometric parameters and suddenly you switched to the classification model with Faster RCNN (for detection) + CNN (for\nclassification). I would describe it with more detail especially because your Multi modal model is based on the use of\ndifferent image modalities and MRI images are only used in this step. Thus I would give more emphasis to this part of the\nmodel.", "reviewer_2": "The paper posits an innovative cascade-ensemble deep learning framework as a solution for diagnosing cervical\nspondylosis, presenting a novel methodology aimed at addressing the diagnostic challenges associated with this condition.\nThe authors' strategy, involving a cascading architecture for vertebral body detection followed by a degenerative diagnosis\nand further bolstered by an ensemble of multiple degenerative indicators within a multi-task learning framework, stands out\nas a sophisticated approach with the potential to navigate the complexities of cervical spondylosis diagnosis effectively.\nNevertheless, several critical concerns merit attention:\nUpon review of the manuscript, a detailed description or presentation of the proposed cascade-ensemble deep learning\nframework is conspicuously absent. The lack of mathematical formulation and detailed architecture for the proposed\nensemble deep learning algorithms significantly impedes understanding and evaluation of the purported methodology.\nWith the existence of over 100 ensemble deep learning (EDL) algorithms documented in the literature, the authors' oversight\nin engaging with relevant existing studies is a notable gap. A comparative analysis or discussion contextualizing their\nproposed framework within this extensive body of work would significantly enhance the manuscript's contribution to the field.\nThe manuscript suggests a particular sensitivity of the model to certain indicators, such as the SVA value and\nspondylolisthesis distance. However, there is a lack of discussion regarding the model's performance variability across\ndifferent conditions and its potential vulnerability to overfitting on these indicators. Addressing overfitting is crucial to ensure\nthe model's applicability and effectiveness in diverse real-world scenarios, where patient presentations can vary widely.\nWhile claims of the model's performance being comparable to that of senior radiologists and clinical doctors are made, there\nis an evident deficiency in details regarding the clinical validation process. Specifically, the manuscript does not elucidate\nhow the deep learning model's diagnostic outcomes were compared with longitudinal patient outcomes or their integration\ninto existing clinical workflows. The true value of a diagnostic tool is not solely in its mimicry of expert opinions but also in its\ndemonstrable impact on patient management and outcomes.\nTo address these concerns, it is strongly recommended that the authors elucidate the mathematical formulation of their\nproposed framework and undertake a comparative analysis with recent ensemble deep learning algorithms. Additionally,\nreference to comprehensive reviews such as \"Ensemble deep learning: A review\" could provide valuable insights and\ncontext for refining and substantiating their contributions to the field."}, "author_response": "Response to the Reviewers’ Comments\nWe appreciate the thorough evaluation of our study by the reviewers and editors.\nIn the revised version of our manuscript, we address essentially all of the points\nraised by the reviewers, and we think the resulting study is much improved. Specific\nrevisions and responses to each comment are provided in detail below.\nPreliminary notes:\n• Section,andpagerefertothedoublecolumnversionoftherevisedmanuscript.\n• We use two complementary conventions to refer to tables (or figures): Table\n1 refers to TABLE 1 of the double column version of the revised manuscript,\nwhile mark like Table R1 refers to this response letter.\n• References are re-organized (including the reference numbers in the com-\nments of reviewer) in this response paper, and all of them are listed in the end\nof the response.\n• Better review in color. The texts in red, black and blue are the comments\nof reviewers, our response and revised text (in double column manuscript),\nrespectively.\nReviewer #1\nSummary Comment. The manuscript presents a new deep learning model to auto-\nmatically assess cervical spondylosis. I enjoyed reading the manuscript but I have a\nfew concerns.\nThanks for acknowledging this paper. And we will try to address your concerns\nin the following response.\nQ1. At the end of the Introduction, there is the sentence that describes your model\nas multi-modal, multi-task, and ensemble. I am not sure if you can use these terms.\nMulti-modal means that you use different data modalities (images, clinical data,\nomics, etc.) and combine them to improve your prediction and have a full picture\nof the patient’s health status. In your case, you have two image modalities but you\nuse them independently (model for radiographs and model for MRI). Am I missing\nsomething? Could you clarify? It makes sense to use two image modalities but you\nare not combining them. In your case, the two modalities are treated separately so\nthere are two stand-alone models.\nResponse 1. Thanks for the useful comments. We believe our method is a\nmultimodal frameworks since we take multimodal input to provide a complementary\nreports for the doctors. And different indicators in our report are simultaneously\nlearned with multi-task manner. All the indicators are ensembled in the reports.\nAs the reviewer pointed out, we use two image modulates, including X-ray and\nMR images, both of which examinations are universally regarded as one of the most\nimportant imaging studies for cervical spondylosis, from the point view of worldwide\ncervical spine research. For instance, a German team conducted an in vivo study\non the biochemical effectiveness of imaging technologies such as X-rays and MRIs in\nhuman cervical degenerative disc diseases [29]. A Chinese team completed a long-\nterm follow-up study on cervical disc arthroplasty based on X-rays and magnetic\nresonance imaging [28]. Another team conducted a retrospective cohort study on\nthe application of X-rays and MRIs in cervical spondylotic myelopathy [27].\nWe agree that in most previous works the multi-modal data are combined into\nsome common feature space to predict a definitive prediction or conclusion. How-\never, in contrast to those works combining multi-modal features together,\nthe cervical spondylosis requires the collaboration of multiple modalities\nto provide richer indicators for clinical diagnosis, due to the diversity\nof clinical appearances. So, according to the particular requests from clinical\ndoctors, our multimodal fusion framework consists of several stages: 1) Stage one\n(indicators detection), providing more available and accurately detected indicators\nfor doctors; 2) Stage two (degeneration classification), classifying the degeneration\nof different indicators. And all the results will be provided for the clinical doctors\n2\nto generate operation plan.\nLet us illustrate with a representative case of cervical spondylosis for instance: a\n37-year-old male patient presents with neck pain, bilateral upper limb pain, numb-\nness, and tetraparesis for two years. The patient’s cervical spine images are shown\nas below. This patient exhibits cervical kyphosis, spinal canal stenosis, and verte-\nbral segmental instability on cervical X-rays (see Fig. R1 a-c). Meanwhile, cervical\nMRIs reveal multiple typical imaging manifestations of cervical spondylosis, such as\ncervical disc herniation and spinal cord compression with degeneration (see Fig. R1\ne-f. From a doctor’s perspective, cervical X-rays can provide an overview of the\ncervical spine’s morphology, particularly its curvature (kyphosis, instability), and\nthe degree of spinal canal stenosis, which is crucial for planning the surgical ap-\nproach and comparing preoperative and postoperative conditions. MRIs, on the\nother hand, offer insights into the location and extent of spinal cord and nerve root\ncompression, determining the diagnostic type of cervical spondylosis, the necessity\nof surgery, and the surgical plan. Meanwhile, the patient’s kyphosis Cobb angle\non X-ray, spinal canal stenosis, and vertebral segmental instability are consistent\nwith the areas of spinal cord compression and signal changes observed on MRI, all\nlocated in the C3-6 segments, thus forming a mutual confirmation between X-ray\nand MRI findings. This consistency of multimodal mutual confirmation serves as\ncrucial evidence required by clinicians for diagnosis and treatment.\nIn conclusion, we designed our multimodal diagnostic framework of cervical\nspondylosisfromtheclinicalrequestsofdoctors. Typicalcervicalspondylosisindica-\ntors, such as Cobb angle, SVA value, intervertebral foramen stenosis, spinal stenosis,\nare required to be explicitly provided for diagnosis. Different cervical spondylosis\nindicators are best suited to be acquired through different data modalities. So we\ndesign our framework with multimodal collaboration instead of fusion to one over-\nall result. The key contribution of our work to doctors lies not only in providing\nresults from a multimodal perspective, but also in offering the potential for mu-\ntual confirmation among different indicators through a multitask learning approach.\nSpecifically, the detection of indicators such as Cobb angle and SVA value based on\nX-rays is aimed at the analysis of cervical curvature degeneration, while the detec-\ntion of indicators such as intervertebral disc herniation, spinal canal stenosis, and\nintervertebral foraminal stenosis based on MRI reflects the direct compression on\nthe spinal cord. By leveraging the advantages of both modalities and analyzing from\nboth the curvature and compression perspectives simultaneously, we can not only\nmore reliably detect the true causes of cervical spondylosis but also achieve mutual\nverification, providingdoctorswithamorereliablebasisforclinicaldecision-making.\nQ2. Regarding multi-task: why is the model multi-task? You predict coordi-\nnates and from your predictions, you compute some measures. My understanding of\nmulti-task is that you have multiple tasks (i.e. localization + classification) and you\noptimize both to improve the prediction of the main task. Could you please clarify?\nX-ray, showing cervical kyphosis and spinal canal stenosis. (b-c)Cervical spine X-\nray in hyperextension and hyperflexion positions, showing segmental instability at\nC4-5 level. (d) Sagittal view of cervical MRI, showing spinal canal stenosis at C3-\n6 levels, with C3-4 disc herniation compressing the spinal cord, resulting in spinal\ncord degeneration and hyperintense signals. (e) Axial view of cervical MRI, showing\nspinal canal stenosis and disc herniation at C3-4 level, compressing the spinal cord\nand causing changes in its morphology.\nResponse 2. The reason why this proposed model denoted as \"multi-task\" is that\nwesimultaneouslyoptimizemultipledegenerativefactorsincludingCobbangle, SVA\nvalue, spondylolisthesis angle, etc. Each single degenerative factor is trained with\na separate loss. The loss between the ground truth labels and the predicted values\nprovide a gradient for updating the parameters of the backbone network, which\npredicts the vertebral key points coordinates.\nThanks for your detailed suggestions about the clarification of multi-task setting.\nAlthough the prediction of these degenerative factors is based on the coordinates\nof vertebral key points, we optimize the coordinates using the loss of each task,\nin addition to optimizing the coordinates themselves. Because our further goal is\nto detect the degenerative factors, which are crucial for making a diagnosis of a\ncervical spondylosis. The vertebra coordinates are used as the intermediate results,\nwhich could be regarded as the pre-processing for the multi-task learning of those\n4\ndegenerative factors.\nAs the reviewer suggested, a typical type of multi-task learning could be \"lo-\ncalization + classification\", like Faster-RCNN [6] shown in R2 a. In the Faster\nR-CNN, the image goes through the backbone convolutional network, obtaining the\nintermediate feature map, then goes through two branches, one of which generates\ncoordinates for localization, and the other generates scores for classification. The\nground truth label for localization and classification serve as supervision, and the\nloss gradients of the supervision is calculated as back propagation, which is used to\nupdate the parameters of the backbone network. The training process on these two\ntasks together improves the performance of the backbone network. In summary, we\nagree to the reviewer that the core of \"multi-task learning\" is to \"learn\" in different\ntasks.\nSimilar to multi-task learning in R2 a, our goal is also to learn a neural network\nmodel in multiple degenerative factors diagnose tasks (see R2 b). In our framework,\nthe image goes through the backbone convolutional network, obtaining the coor-\ndinates of cervical vertebral key points. Based on these coordinates, we compute\ndifferent degenerative indicators, such as Cobb angle, SVA value, spondylolisthesis\nangle, etc. Then we use the ground truth values of these indicators as supervision,\nandbackpropagatethegradientsoflosstoimprovetheperformanceofthebackbone\nnetwork. The training process on every single degenerative indicator is considered a\ntask, and these different tasks can jointly improve the performance of the vertebral\nkey points prediction module.\nQ3. Regarding ensemble: similar comment. Your model predicts different out-\ncomes (different parameters) while in ensemble learning you have many learners\nthat predict the same outcome and the very final outcome is predicted by averag-\ning/voting/.... the predictions by the learners.\nResponse 3. Thanks for your useful suggestions. We agree that in most pre-\nvious work, ensemble learning methods have multiple learners and obtain the final\nresults through averaging or voting the predictions by these learners. However, in\nthe diagnosis of cervical spondylosis, it is not simply about averaging the results of\nmultipledegenerativeindicatorstocompletethediagnosis. Doctorsmakejudgments\nby referring to multiple indicators together. Therefore, in order to assist doctors in\ndiagnosing cervical spondylosis, at the patient level, we ensemble multiple degener-\native indicators to make a diagnosis of cervical spondylosis.\nWe would like to provide a detailed description of how clinical doctors diagnose\ncervical spondylosis in practice. The diagnosis of cervical spondylosis is a com-\nprehensive process that integrates clinical manifestations with radiological features.\nFeaturemap\nloss\nLocalization Coordinates\nFasterR-CNN:\nloss\nClassification Labels\nCNN\nKey\npoints\ncoordinates\na\nloss\nb\nCobb angle Labels\nloss\nSVA value Labels\nOurs:\nloss\nSpondylolisthesis\nLabels\nangle\n…\nFigure R2: Multi-task setting of the Faster R-CNN and our method. (a)Multi-task\nlearning framework of the Faster R-CNN. (b)Multi-task learning framework of our\nproposed method.\nCervical spondylosis primarily arises from static compression of the cervical spinal\ncord or nerve roots (such as cervical disc herniation, cervical spinal canal stenosis, or\nneural foramen stenosis) or dynamic instability (such as vertebral segmental insta-\nbility), leading to symptoms such as pain, numbness, or weakness in the extremities.\nThe common types of cervical spondylosis include radiculopathy (characterized by\npain and numbness caused by nerve root compression) and myelopathy (character-\nized by weakness and numbness due to spinal cord compression), with occasionally\na mixed type where both are present.\nThe development of cervical spondylosis is often associated with multiple fac-\ntors, including cervical disc herniation, developmental stenosis of the cervical spinal\ncanal, neural foramen stenosis, bone spur formation, ossification of the posterior lon-\ngitudinal ligament of the cervical spine, cervical segmental instability, and cervical\nkyphosis. Each of these causative factors exhibits distinct radiological manifesta-\ntions, for instance, cervical disc herniation is clearly visualized on cervical magnetic\nresonance imaging (MRI), developmental stenosis of the cervical spinal canal can be\nassessed by measuring the Pavlov ratio on X-ray, cervical segmental instability is\nmanifested by anterior and posterior sliding of the cervical spine in over extension\nand over flexion positions, and cervical kyphosis can be determined by SVA value.\nThese radiological features not only aid in diagnosis but also guide the formula-\ntionoftreatmentplans. Eachimagingfeatureindicatesaspecificsurgicalprocedure,\n6\nsuchascervicaldischerniationrequiringanteriorsurgery,cervicalcanaldevelopmen-\ntal stenosis requiring posterior surgery, nerve root canal stenosis treated anteriorly,\nosteophyte formation treated anteriorly, ossification of the cervical posterior longitu-\ndinal ligament requiring posteriorly, cervical segmental instability treated anteriorly,\nand cervical kyphosis managed both anteriorly and posteriorly.\nIt is important to note that the diagnosis of cervical spondylosis does not require\nthe patient to exhibit all possible clinical manifestations or radiological abnormali-\nties simultaneously. In fact, a single typical clinical symptom accompanied by the\ncorresponding radiological feature is sufficient for diagnosis. For example, cervical\ndisc herniation accompanied by pain can be diagnosed as radiculopathy, while devel-\nopmental stenosis of the cervical spinal canal with weakness indicates myelopathy.\nFromtheabovedescription, wecanseethateachdegenerativeindicatoriscrucial\ninthediagnosisofcervicalspondylosis. Consequently,ratherthandesigningmultiple\nlearners to predict the same outcome, we train a model with multiple outputs,\nensemble these outputs, and feedback them to the doctors. Then the doctors can\ndeduce the final diagnosis by integrating these outputs with the patient’s clinical\nsymptoms.\nQ4. The general comment is to try to clarify/explain better all the concepts you\nmentioned (multimodal/multitask/ensemble).\nResponse 4.\nMulti-task. Thepurposeofdesigningmulti-tasklearningwithdifferentindicators\nis to hope that during the learning process, different indicators pointing to the same\ndegenerative area can corroborate each other. For example in Fig.R3, consistency\namong indicators is illustrated as follows:\n\"To some extent, C3-C6 Cervical disc herniation is a direct cause of Spinal canal\nstenosis. As the degenerative changes in the cervical spine of the patient progress,\nmultiple discs at C3-C6 gradually undergo herniation, reducing the effective space\nwithin the spinal canal, thereby forming cervical spinal stenosis and ultimately lead-\ning to compression of the cervical spinal cord or nerve roots. At the same time,\nSpinal canal stenosis is also a manifestation of the compression of the spinal cord by\nC3-C6 Cervical disc herniation, and the two conditions can corroborate each other.\"\nOur model is considered multi-task because it simultaneously optimizes multi-\nple degenerative factors, including Cobb angle, SVA value, spondylolisthesis angle,\netc. Each degenerative factor is trained with its own loss function, and the loss be-\ntween the ground truth labels and predicted values provides a gradient for updating\nthe parameters of the backbone network, which predicts the vertebral key points\n7\ncoordinates.\nAsweexplainedearlier, ourmodelissimilartothetypicalmodelusingmulti-task\nlearning approach, [6], the architecture also can be seen in Fig.R7. In the Faster\nR-CNN, the localization branch and the classification branch are jointly trained.\nThe learning goal on these two tasks is to improve the performance of the backbone\nnetwork, whichextractthefeaturemapfromtheimages. Inourmodel, everybranch\npredicts a degenerative factor, and the supervised training process on these tasks\nimproves the performance of the vertebral key points coordinates prediction module.\nParticularly, wehaveincludedmoredescriptionsofmulti-tasklearningasfollows:\nMulti-task learning. All above metrics Q = [Q ,Q ,Q ,Q ,Q ]\nCA SD SA SDVB SDVC\nare jointly participated in multi-task training and optimization for X-rays. And\nthis multi-task framework is implemented as the extension of Faster R-CNN model\n[6]. In Faster R-CNN model, two tasks of classification and localization are jointly\nlearned as two branches after several shared CNN layers. In our implementation\n(see Fig.R6), each metric in Q is implemented as a branch as the follows:\nM\n(cid:88) (cid:0) ¯ (cid:1)\nL = λ MAE Q ,Q\nmulti−task m m m\nm=1\nMulti-modal. We believe our method is a multimodal frameworks since we take\nmultimodal input to provide a complementary reports for the doctors. And different\nindicators in our report are simultaneously learned with multi-task manner. All the\nindicators are ensembled in the reports. First, we would like to clarify the terms\nquestioned by the reviewer.\nWe agree that in most previous works the multi-modal data are combined into\nsome common feature space to predict a definitive prediction or conclusion. How-\never, in contrast to those works combining multi-modal features together,\nthe cervical spondylosis requires the collaboration of multiple modalities\nto provide richer indicators for clinical diagnosis, due to the diversity\nof clinical appearances. So, according to the particular requests from clinical\ndoctors, our multimodal fusion framework consists of several stages: 1) Stage one\n(indicators detection), providing more available and accurately detected indicators\nfor doctors; 2) Stage two (degeneration classification), classifying the degeneration\nof different indicators. And all the results will be provided for the clinical doctors\nDiagnostic Report\nPatient Information\nPatient name: Anonymous\nAge: 38\nGender: female\nChief complaint: Neck pain for 5 months. Numbness on the outside of\nthe right forearm, right thumb, index finger, median for 1 month.\nMedical Imaging\nCobb angle: 12.07°\ncervical spine curvature: normal\nSVA：6.05mm\ncervical spine sagittal plane: normal\nPavlov:\nC3: 0.85,C4:0.71,C5:0.72,C6:0.84,C7:0.79\ndevelopmental cervical:C4,C5 stenosis\nSpondylolisthesis distance:\nC2-C3:1.86mm,C3-C4:3.27mm,C4-C5:3.34mm\nC5-C6:7.23mm,C6-C7:4.88mm\nSpondylolisthesis angle:\nC2-C3:-1.22°,C3-C4:-8.99°,C4-C5:-9.6°\nC5-C6:-21.44°,C6-C7:-6.72°\nvertebral stability: C5-C6unstable\nIntervertebral disc:\nC3-C6 herniation\nspinal canal: stenosis\nright foramen: normal\nleft foramen: normal\nAI Diagnostic：\nCervical spine degeneration\nC3-C6 Cervical disc herniation\nSpinal canal stenosis\nFigure R3: A visualized diagnostic report of a real case.\n9\nto generate operation plan.\nIn conclusion, we designed our multimodal diagnostic framework of cervical\nspondylosisfromtheclinicalrequestsofdoctors. Typicalcervicalspondylosisindica-\ntors, such as Cobb angle, SVA value, intervertebral foramen stenosis, spinal stenosis,\nare required to be explicitly provided for diagnosis. Different cervical spondylosis\nindicators are best suited to be acquired through different data modalities. So we\ndesign our framework with multimodal collaboration instead of fusion to one overall\nresult.\nEnsemble. Our model is based on the concept of ensemble learning, which means\nwecombinemultipledegenerativeindicatorstodiagnosecervicalspondylosis. Unlike\ntraditional methods that simply average or vote on predictions, our model reflects\nthe way doctors make judgments by considering multiple indicators together.\nThe diagnosis of cervical spondylosis is a comprehensive process that integrates\nclinicalmanifestationswithradiologicalfeatures. Thedevelopmentofcervicalspondy-\nlosis is often associated with multiple factors including cervical disc herniation, de-\nvelopmental stenosis of the cervical spinal canal, neural foramen stenosis, bone spur\nformation, ossification of the posterior longitudinal ligament of the cervical spine,\ncervical segmental instability, and cervical kyphosis. Each of these factors manifests\nwith different radiological presentations, therefore, pathogenic factors can be in-\nferred from the X-ray images, which helps in diagnosis and formulation of treatment\nplans.\nRather than designing multiple learners to predict the same output, our model\nwas trained with multiple outputs. Then we ensemble these outputs and provide\nfeedback to the doctors, allowing them to deduce the final diagnosis by integrating\nthese outputs with the patient’s clinical symptoms. This approach reflects the com-\nplex and integrated nature of diagnosing cervical spondylosis and aims to provide\ndoctors with valuable assistance in making accurate diagnoses.\nResults\nQ5. In the first paragraph, it is written ’1500 patients’. Then, In the paragraph after\nthere is a reference to 1151 cases. I am missing the consistency of the numbers.\nPlease clarify. Also, you only mentioned radiographs and not MRI data.\nResponse 5. Thanks for the useful comments. In the first paragraph, the \"1500\npatients\" refer to the number of cases collected by the hospital. The reason why\nthe number is less than 1500 is that some cases were excluded due to inappropriate\nlabeling (missing some vertebral bodies or spinous processes). However, in the\n10\nend, there were 230 test cases with consistent results across MRI stenosis, MRI\nintervertebral discs, and X-rays. To eliminate ambiguities in data description, we\nhave consolidated all relevant data-related descriptions into the following paragraph:\n\"This retrospective study collected cervical spine X-ray and MRI scans of 1500\npatients (mean age ± standard deviation: 54 years ± 10.28, men: 909, 60.60%)\nwith cervical spondylosis the Third Hospital of Peking University from January\n2016 to December 2018. During the preparation of data, some cases were excluded\ndue to inappropriate labeling, such as missing vertebral bodies or spinous processes,\nresulting in a total of 1151 cases being used. Data collection was pre-approved\nby the Institutional Review Board and excluded minors as well as patients with\ninstrumentation or other medical conditions (e.g., spinal tumors, infections, trauma,\nor scoliosis). They were annotated by three radiologists and randomly divided into\na training set (693), a validation set (228) and a testing set (230). Among these,\nthe results from MRI stenosis, MRI intervertebral discs, and X-rays were consistent.\nAll data were manually labeled by radiologists and validated against each other.\nThe labeling information was all by the four-point method of vertebrae in lateral\nand flexion-extension positions on cervical spine radiographs. Each radiologist was\nblinded to patient demographics and clinical history. The dataset contained one\ncervical spine radiograph of each patient in three positions of neutral, flexion, and\nextension in the lateral position.\"\nVertebral detection X-ray images\nQ6. How is the error computed? Is it the Euclidean distance?\nResponse 6. The errors of key point coordinates are calculated with the concept\nofEuclideandistance, andtheformulaforMeanAbsoluteError(MAE)isasfollows:\n1 (cid:113)\nMAE = (cid:80)N (x −x¯)2 +(y −y¯)2 (1)\nN i=1 i i i i\nwhere N is the number of key points,(x ,y ) represents the actual coordinate of\ni i\nthe i−th key point, and (x¯,y¯) represents the estimated coordinate of the i−thkey\ni i\npoint.\nPage 8 C2-7 SVA paragraph\nQ7. Please check the references to the figures in all Results paragraphs. Some of\nthem are wrong.\nResponse 7. Thanks for your correction. We have modified \"sub-figure Fig. 2 a\"\nto \"sub-figure Fig. 3 a\", \"sub-figure Fig. 2 b\" to \"sub-figure Fig. 3 b\", \"sub-figure\nIntervertebral Degeneration Analysis on MR images\nQ8. I do not get how you can say that your model outperformed the clinical assess-\nment as the ground truth to train the model was generated from clinicians. How was\nthe ground truth computed? Average of annotations? Voting? Consensus? Please\nclarify how the ground truth was generated.\nResponse 8. In general, our ground truth is obtained by reaching a consensus\namong experts and junior doctors. The details are as follows: Regarding the delin-\neation of X-ray imaging components, specifically the identification of vertebral body\nlandmarks and the delineation of spinous processes, the gold standard protocol man-\ndates that a primary physician performs the initial marking. This is followed by a\nsecondary physician reviewing and verifying the markings, with revisions made as\nnecessary to achieve consensus. In the context of MR imaging, the delineation of the\ncentral spinal canal and the grading of intervertebral foramen stenosis is conducted\nby two junior physicians, if they have divergent opinions, an expert will make the\nfinal determination, which is considered as the gold standard.\nIntervertebral foramen stenosis\nQ9. In the last sentence, you refer to a statistical analysis to evaluate intervertebral\ndisc assessment. I did not find a reference to this analysis in the Methods and I do\nnot understand what kind of analysis it is. There is also a typo after the ’3’.\nResponse 9. Thank you for your question. Previously, we investigated a rough\nanalysis of the time cost for cervical MRIs of a few cervical spondylosis patients,\nand due to space limitations, we did not display the specific results. This time,\nwe have made more detailed additions. We collected data from 282 patients for a\ntime cost survey and analysis. The results showed that the average time to assess\nintervertebral degeneration was 2.93 minutes (the time cost is calculated from the\nmoment the doctor opens the patient’s images to the time the images are closed.),\nas detailed in Fig.R4. Regarding the typo you mentioned, I noticed that it should\nbe \"minutes\" after \"3.\" Thank you for pointing it out, and we will correct this error\nin the revised manuscript. Once again, thank you for your valuable feedback. We\nwill ensure that all content in the paper is clear and accurate.\nThe Patient level Diagnostic Analysis\nQ10. Check again the Figures references.\nResponse 10. Thanks for your correction. We have rectified the wrong parts.\nDiscussion\nQ11. I am not sure that you can compare your work with the works you mentioned\nin the discussion because they are computing the Cobb angle on the frontal plane for\nthe full spine while you are just using the cervical region. In your case, it could be\neasier to get better results because your Cobb is always calculating between C2 and\nC7 while the Cobb angle is the maximum angle computed on the frontal plane.\nResponse 11. The spine of the human body is divided into the cervical spine (C1-\n7), thoracic spine (T1-12), lumbar spine (L1-5), and sacral spine (S1-5). Each type\nof vertebrae is similar in shape but has unique anatomical characteristics. In the\nsagittal plane, the normal spine has cervical lordosis, thoracic kyphosis, lumbar lor-\ndosis, and sacral kyphosis. In the coronal plane, the overall spine is vertical without\ncurvature. Spinal diseases in spinal surgery can generally be divided into degener-\native diseases (such as cervical spondylosis), spinal deformities (such as scoliosis),\nspinal tumors, spinal trauma, and spinal infections. Degenerative spinal diseases are\nthe most commonly seen in clinical practice.\neven determine the surgical approach and segment. For example, patients with\nsevere cervical spondylosis may experience typical symptoms of cervical spondylosis,\nsuch as pain, numbness, and weakness, as the anterior convexity of the cervical\nspine in the sagittal plane may become straight or even posterior convexity due to\nprolonged lowering of the head[7]. And we need to measure the angle changes in the\nsagittal plane of the cervical spine to determine the surgical approach and effect[8].\nSimilarly, for thoracolumbar scoliosis deformities, there is often a change in spinal\nangle in the coronal position, and we need to determine the surgical range based on\nthe size and extent of the scoliosis angle.\nTherefore, in spinal surgery, we use the concept of Cobb angle to describe the\nalignment and curvature angles of the spine. This concept can describe both the\nsagittalandcoronalanglesofthespine, basedonthemorphologicalcharacteristicsof\ndifferent types of spinal diseases. As mentioned earlier, for cervical spondylosis, the\ncommon change in spinal angle is a decrease in the anterior convexity angle in the\nsagittal plane, while for thoracolumbar scoliosis, the common change is an increase\nin the angle in the coronal plane [9, 10]. Therefore, the definition of \"cervical Cobb\nangle\" often refers to the angle between the extension line of the lower endplate of\nthe C2 vertebra and the extension line of the lower endplate of the C7 vertebra in\nthe sagittal plane[11]. If the patient’s X-ray shows unclear display of C7 vertebrae,\nC6 vertebrae can be used instead[12]. Similarly, Cobb angle has also been applied\nand studied on cervical sagittal CT[13].\nBased on our team’s literature search, we have found that there are relatively few\npapers studying artificial intelligence in cervical spine imaging (such as Cobb angle).\nClinically, whether it is the cervical spine, lumbar spine, or scoliosis, and whether\nit is in the sagittal plane or coronal plane, the method for measuring the Cobb\nangle is generally consistent, and the manual measurement steps and methods are\nalmostthesame, thusmakingthemcomparable. Althoughthecervicalvertebraeare\nslightly smaller than the thoracolumbar vertebrae, the manual measurement errors\nare basically the same. The difference in accuracy of artificial intelligence models\ndepends more on the algorithm construction of the model itself. When referring\nto degenerative diseases of the cervical or lumbar spine, the Cobb angle refers to\nthe sagittal plane, while in cases of thoracolumbar scoliosis, it often refers to the\ncoronal plane[14], and the two are not contradictory. In the revision of this article,\nwe have included specific literature related to the \"cervical Cobb angle\" to increase\nthe relevance and specificity of the literature cited in this article to cervical spine\nartificial intelligence as follows:\n14\n\"Therefore, in spinal surgery, the concept of Cobb angle is used to describe the\nalignment and curvature of the spine. This concept can be applied to describe\nangles in both the sagittal and coronal planes, depending on the morphological\ncharacteristics of different types of spinal diseases. For cervical spondylosis, the\ncommon change in spinal angle is a reduction in the lordotic angle in the sagittal\nplane; whereas for thoracolumbar scoliosis, the common change is an increase in\nangle in the coronal plane [10]. Therefore, the definition of \"cervical Cobb angle\"\noften refers to the angle between the extension line of the lower endplate of the\nC2 vertebra and the extension line of the lower endplate of the C7 vertebra in the\nsagittal plane[11]. If the patient’s X-ray shows unclear display of C7 vertebrae, C6\nvertebrae can be used instead[12]. Similarly, Cobb angle has also been applied\nand studied on cervical sagittal CT[13].\"\nMethods\nQ12. Since C2 is a vertebra that is quite different from the other vertebrae (it does\nnot have a clear upper endplate), how did you define the upper endplate corners?\nResponse 12. Firstly, the measurement of the cervical Cobb angle uses the lower\nendplate of the C2 vertebra and does not involve the upper endplate of the C2\nvertebra. Secondly, anatomically, there is no such concept as the upper endplate\nof the C2 vertebra. It is only when we calculate the SVA value that we have a\nconceptually virtual upper endplate in imaging.\nThe primary objective of delineating the C2 vertebra is to identify the centroid\nof the C2 vertebra, which is essential for the measurement of the Sagittal Vertical\nAxis (SVA) value. Therefore, we designate the portion of the C2 vertebra exclud-\ning the odontoid process, where it intersects with the transverse processes, as the\nupper endplate corners. As illustrated in the Fig. R5, this method is utilized in the\nliterature to ascertain the center of the C2 vertebra[5, 30, 31]. We also follow this\nmethod to define the upper end plate corners.\nQ13. Specify what is Lgroup and how it is computed.\nResponse 13. In the general keypoint detection model, the loss function requires\nan L term. This term is designed to discern distinct instances within a sin-\ngroup\ngle image by minimizing intra-class distances and maximizing inter-class distances.\nHowever, in the context of cervical spine radiography, there is typically a single in-\nstance depicted, thus our loss function L excludes a clustering loss component.\nours\nThis exclusion can be conceptualized as the multiplication of the clustering loss term\nby a parameter that is nullified to zero.\nTo eliminate ambiguity in this section, we have removed this representation.\n2−7\nQ14. Line 20 very last line: I do not get the sentence. What do you mean by\n’Particularly for the medical images,.....’.\nResponse 14. Compared to natural image processing, key point detection in\nmedicalimagesrequireshigherprecisiontoobtainmoreaccuratemetriccalculations.\nSowegeneratetheheatmaponthehighestresolutionfeaturemap,whichisbeneficial\nfor preserving more positional information.\nDegeneration Diagnosis in Multi-task Learning\nQ15. As outlined before, I am not sure that you can talk about multi task approach.\nPlease explain. - Page 24 last sentence before Evaluation metrics paragraph: The\nsentence seems out of context. You described all the geometric parameters and sud-\ndenly you switched to the classification model with Faster RCNN (for detection) +\nCNN (for classification). I would describe it with more detail especially because your\nMulti modal model is based on the use of different image modalities and MRI images\nare only used in this step. Thus I would give more emphasis to this part of the\nmodel.\nResponse 15. Thank you for your useful suggestions. We have provide a more\ndetailed explanation of the definition of multi-task.\nAs the reviewer suggested, the key of our multi-modal model is to detect differ-\nent geometric indicators (parameters). The previous ’Page 24 last sentence before\nAll above metrics Q = [Q ,Q ,Q ,Q ,Q ] are jointly participated\nCA SD SA SDVB SDVC\ninmulti-tasktrainingandoptimizationforX-rays. Andthismulti-taskframeworkis\nimplemented as the extension of Faster R-CNN model [6]. In Faster R-CNN model,\ntwo tasks of classification and localization are jointly learned as two branches after\nseveral shared CNN layers. In our implementation (see Fig.R6), each metric in Q is\nimplemented as a branch as the follows:\nM\n(cid:88) (cid:0) ¯ (cid:1)\nL = λ MAE Q ,Q\nmulti−task m m m\nm=1\nThe diagnosis of intervertebral disc degeneration on MRI could also implemented as\nabranchafterseveralsharedCNNlayers. However,consideringthevisualdifferences\nbetween X-ray and MRI images, we separately train a different CNN model for MRI\nimages with the classification loss function of intervertebral disc degeneration levels\n(see Fig.R7).\nReviewer #2\nSummary Comment. The paper posits an innovative cascade-ensemble deep learn-\ning framework as a solution for diagnosing cervical spondylosis, presenting a novel\nmethodology aimed at addressing the diagnostic challenges associated with this con-\ndition. The authors’ strategy, involving a cascading architecture for vertebral body\ndetection followed by a degenerative diagnosis and further bolstered by an ensemble\nof multiple degenerative indicators within a multi-task learning framework, stands\nout as a sophisticated approach with the potential to navigate the complexities of cer-\nvical spondylosis diagnosis effectively. Nevertheless, several critical concerns merit\nattention:\nThank you very much for acknowledging the novelty, effectiveness, and potential\nof our paper in diagnosing cervical spondylosis through the proposed deep learning\nframework. And we will try to address your concerns in the following response.\nQ1. Upon review of the manuscript, a detailed description or presentation of the\nproposed cascade-ensemble deep learning framework is conspicuously absent. The\nlack of mathematical formulation and detailed architecture for the proposed ensemble\ndeep learning algorithms significantly impedes understanding and evaluation of the\npurported methodology.\nResponse 1. Thanks for your suggestion. We have provided a further detailed\ndescription of the structure of our proposed model, and the entire structure of the\n17\nmodel is shown in R6 and R7.\nThe model for processing X-ray images is divided into two stages. The first\ndetection stage detects the coordinates of cervical vertebral key points, while the\nsecond prediction stage predicts multiple degenerative indicators. Our detection\nmodule extracts the feature map from the input images and then generates the\nheatmap on the highest resolution feature map, inspired by the HRNet[4, 2] and\nHigherHRNet[3]. The detection module is a parallel architecture that consists of\nseveral branches, where one branch focuses on high-resolution features, while others\nprogressivelydownsampletheinputtocreatelow-resolutionrepresentations(seeR6).\nThe high-resolution branch consists of several residual blocks that utilize 3x3 con-\nvolutions, batch normalization, and ReLU activation. Alongside the high-resolution\nbranch, there are low-resolution branches at different scales (1/2, 1/4 the origi-\nnal resolution), and the downsampling is implemented by the strided convolutions.\nThe architecture also employs a fusion mechanism to combine high-resolution and\nlow-resolution feature maps, using upsampling and convolution operations. After\nobtaining the feature map through the above backbone network, we use deconvo-\nlution upsampling operation and several residual convolutional layers to obtain the\nfinal high resolution heatmaps. Then we use the Differentiable Spatial to Numeri-\ncal Transform (DSNT)[1] to get the mapped pixel coordinates from the normalized\nheatmap prediction, and finally group the keypoints of different instances. Finally\nthe predicted values of various degenerative indicators are calculated from the linear\nlayers based on the coordinates obtained by the DSNT mapping.\nThe model for processing MR Images is built on Faster R-CNN[6]. We detect the\nintervertebral foramen and spinal stenosis using Faster R-CNN model, and diagnose\nthe intervertebral disc degeneration using classifier based on CNN. As shown in R7,\nwe utilize ResNet50[25] and FPN[26] as the backbone network to extract feature\nmaps from the input images. The backbone’s feature map serves as the input to\nthe Region Proposal Network (RPN), which is a fully convolutional network that\nslides over the feature map to propose candidate object bounding boxes. The re-\ngion proposals generated by the RPN are then fed into the Region of Interest(RoI)\npooling layer. This layer takes each proposed region and pools features from the\ncorresponding locations in the feature map generated by the backbone, converting\neach proposed region into a fixed-size feature map, regardless of the proposal’s orig-\ninal size. Then some fully connected layers are used to generate the final outputs,\nconsisting of the bounding box and the classification scores.\nBased on the ensemble model categorization in [15], our work basically falls into\nthecategoryofimplicitensemble,i.e.,mostoftheparametersinthemodelareshared\n18\n(specifically, the backbone part used for vertebral detection), while other parts are\nindependent (namely, the sections for calculating indicators such as Cobb angle and\nSVA value). From a mathematical perspective, the training of shared parameters in\nour implicit ensemble approach is obtained by deriving the loss function using the\nfollowing formula:\nM\n(cid:88) (cid:0) ¯ (cid:1)\nL = λ MAE Q ,Q\nmulti−task m m m\nm=1\nwhere metrics Q ∈ Q, Q = [Q ,Q ,Q ,Q ,Q ] are jointly partici-\nm CA SD SA SDVB SDVC\npated in multi-task training and optimization for X-rays.\nFigure R6: The structure of the model for processing X-ray images, composed of\nthe cervical vertebral key points detection module and the degenerative indicators\nprediction module.\nFigure R7: The structure of the model for processing MR Images, built on the\nFaster R-CNN.\nQ2. With the existence of over 100 ensemble deep learning (EDL) algorithms\ndocumented in the literature, the authors’ oversight in engaging with relevant existing\nstudies is a notable gap. A comparative analysis or discussion contextualizing their\nproposed framework within this extensive body of work would significantly enhance\nthe manuscript’s contribution to the field.\nResponse 2. Compared with the existing various ensemble deep learning algo-\nrithms, our ensemble learning model is specifically designed for the diagnosis of\ncervical spondylosis. Traditional ensemble learning methods such as bagging and\nboosting, have multiple base learners, and obtain the final results by averaging or\n19\nvoting the predictions made by these base learners. However, simply averaging or\nvoting is not suitable for the diagnosis of cervical spondylosis. The diagnosis of cer-\nvical spondylosis is a comprehensive process that integrates clinical manifestations\nwith radiological features. So that every degenerative factor predicted by different\nlearner is crucial for the final result, we ensemble them at the patient level and offer\nthem to assisting doctors in diagnosing cervical spondylosis.\nThe ensemble learning in our method is somewhat similar to the operation mech-\nanism of the decision trees, where each degenerative indicator is a node on the deci-\nsion tree, and the values of multiple indicators can form a path on the decision tree,\nleading to the final diagnostic conclusion. In this way, the complex high-dimensional\nproblem of diagnosing cervical spondylosis can be transformed into a simple low-\ndimensional decision.\nQ3. The manuscript suggests a particular sensitivity of the model to certain in-\ndicators, such as the SVA value and spondylolisthesis distance. However, there is a\nlack of discussion regarding the model’s performance variability across different in-\ndicators and its potential vulnerability to overfitting on these indicators. Addressing\noverfitting is crucial to ensure the model’s applicability and effectiveness in diverse\nreal-world scenarios, where patient presentations can vary widely.\nResponse 3. In real-world application scenarios, indicators such as Cobb angle,\nSVAvalue, Pavlovvalue, etc, arewidelyusedinpreoperativeevaluationandsurgical\ndesign. The above indicators are not only suitable for the Chinese population,\nbut also the consensus developed by the world spine surgery community (such as\nthe European AOSpine Association and the American NASS Association) in recent\ndecades. In the research reports of both the Asian population and the European and\nAmerican population, preoperative evaluation and surgical design can be carried out\nthrough the above indicators, which is widely recognized and consistent.\nIn cervical spine surgery, various parameters of cervical lateral X-ray are very\nimportantandmayevendeterminethesurgicalapproachandsegment. Forexample,\npatients with severe cervical spondylosis, due to long-term lowering of the head, the\nanterior convexity of the cervical spine in the sagittal plane may become straight\nor even posterior convexity, resulting in compression of the cervical spinal cord and\ntypical symptoms of cervical spondylosis such as pain, numbness, and weakness[7].\nWe use the Cobb angle in the cervical spine sagittal plane to quantify the changes\nin cervical curvature and determine and evaluate surgical methods and outcomes[8].\nIf the angle of the posterior convex Cobb is too large, it is recommended to perform\nanterior ACDF surgery to correct the curvature[9, 10].\nSimilarly, SVA angle is also an indicator for describing changes in cervical sagit-\n20\ntal curvature. A high SVA value indicates severe kyphosis and is suitable for ante-\nrior orthopedic surgery[16, 17, 18]. For the Pavlov value (sagittal diameter of the\nspinal canal/vertebral body), a value less than 0.75 indicates developmental cervical\nstenosis and is suitable for posterior decompression surgery[19, 20, 21]. For vertebral\ninstability, once it is clear, it implies the need for fixed fusion surgery to solve the\nproblem of vertebral instability[22, 23, 24].\nQ4. While claims of the model’s performance being comparable to that of senior\nradiologists and clinical doctors are made, there is an evident deficiency in details re-\ngarding the clinical validation process. Specifically, the manuscript does not elucidate\nhow the deep learning model’s diagnostic outcomes were compared with longitudinal\npatient outcomes or their integration into existing clinical workflows. The true value\nof a diagnostic tool is not solely in its mimicry of expert opinions but also in its\ndemonstrable impact on patient management and outcomes.\nResponse 4. Thanks for your useful comments.\nFirst, we compared to the junior and senior doctors through expert blind review.\nour method, junior doctors, and senior doctors each provided their own diagnostic\nresults, which were then shuffled and submitted to experts. Since this is a retro-\nspective analysis, experts will judge which result is more accurate in the form of\nmultiple-choice questions based on the full examination results of the cases.\nWe also agree that objective evaluation is not only based solely on the personal\nchoices and judgments of experts, but also in its demonstrable impact on patient\nmanagement and outcomes. Since this is a retrospective study, our experts will\ncomprehensivelydeterminewhichanswerismorereasonablebasedonthesubsequent\ntreatment plans and outcomes of the cases in the hospital record.\nIn addition, it should be emphasized again that experts’ judgments are made\nblindly, ensuring the fairness of the evaluation.\nQ5. To address these concerns, it is strongly recommended that the authors elu-\ncidate the mathematical formulation of their proposed framework and undertake a\ncomparative analysis with recent ensemble deep learning algorithms. Additionally,\nreference to comprehensive reviews such as \"Ensemble deep learning: A review\"\ncould provide valuable insights and context for refining and substantiating their\ncontributions to the field.\nResponse 5. Thanks for your useful suggestion. We have conducted a detailed\nanalysis of the state-of-the-art deep ensemble models according to the suggested\narticle \"Ensemble deep learning: A review\"[15]. and compare our model with\nthese ensemble models. As the reviewer mentioned, the review article, classified\nensemble learning into several categories including bagging, boosting, stacking, neg-\n21\native correlation based deep ensemble models, explicit/implicit ensembles, homoge-\nneous/heterogeneous ensemble, decision fusion strategies based deep ensemble mod-\nels. The ultimate goal of these different types of ensemble methods is to combine\nmultiple base models to obtain better generalization performance.\nCompared to traditional ensemble learning methods such as bagging and boost-\ning, which use multiple base learners and combine their predictions through aver-\naging or voting to obtain final results, we design our model to combine the results\nin a different way. In the case of diagnosing cervical spondylosis, simply averag-\ning the results of multiple degenerative indicators is not sufficient. Instead, doctors\nrely on the combined analysis of multiple indicators to make accurate judgments.\nTherefore, in our approach to assisting doctors in diagnosing cervical spondylosis\nat the patient level, we employ ensemble learning to combine multiple degenerative\nindicators for a more comprehensive diagnosis.\nBased on the ensemble model categorization in [15], our work basically falls into\nthecategoryofimplicitensemble,i.e.,mostoftheparametersinthemodelareshared\n(specifically, the backbone part used for vertebral detection), while other parts are\nindependent (namely, the sections for calculating indicators such as Cobb angle\nand SVA value). However, instead of averaging the results of multiple predictions,\nour work attempts to leverage shared parameters to calculate cervical degeneration\nindicesfromdifferentperspectives, andthenusethesedifferentindicestocross-verify\nand determine the true cause of cervical spondylosis. In our approach, the image\nis passed through the backbone convolutional network to obtain the coordinates of\ncervicalvertebral key points, which couldbeen seenas thelowerlevel features. After\nextracting these shared intermediate features, different degenerative indicators can\nbe obtained through different prediction branch (see Fig.R6).\nFrom a mathematical perspective, the training of shared parameters in our im-\nplicit ensemble approach is obtained by deriving the loss function using the following\nformula:\nM\n(cid:88) (cid:0) ¯ (cid:1)\nL = λ MAE Q ,Q\nmulti−task m m m\nm=1\nwhere metrics Q ∈ Q, Q = [Q ,Q ,Q ,Q ,Q ] are jointly partici-\nm CA SD SA SDVB SDVC\npatedin multi-tasktraining andoptimization forX-rays. Andthis multi-task frame-\nwork is implemented as the extension of Faster R-CNN model [6]. In Faster R-CNN\nmodel, two tasks of classification and localization are jointly learned as two branches\nafter several shared CNN layers.\n[1] Nibali, A., He, Z., Morgan, S. & Prendergast, L. Numerical coordinate re-\n22\ngression with convolutional neural networks. arXiv preprint arXiv:1801.07372\n(2018).\n[2] Wang, J., Sun, K., Cheng, T.et al. Deephigh-resolutionrepresentationlearning\nfor visual recognition. IEEE TRANSACTIONS ON PATTERN ANALYSIS\nAND MACHINE INTELLIGENCE 43, 3349–3364(2021OCT1). URLhttps:\n//doi.org/10.1109/TPAMI.2020.2983686.\n[3] Cheng, B., Xiao, B., Wang, J. et al. Higherhrnet: Scale-aware representation\nlearning for bottom-up human pose estimation (2020). URL https://doi.\norg/10.1109/CVPR42600.2020.00543.\n[4] Sun, K., Xiao, B., Liu, D.et al. Deephigh-resolutionrepresentationlearningfor\nhuman pose estimation (2019). URL https://doi.org/10.1109/CVPR.2019.\n00584. Proceedings Paper.\n[5] Ailon, T., Smith, J. S., Shaffrey, C. I. et al. Outcomes of operative treatment\nfor adult cervical deformity: a prospective multicenter assessment with 1-year\nfollow-up. Neurosurgery 83, 1031–1039 (2018).\n[6] Ren, S., He, K., Girshick, R. B. & Sun, J. Faster R-CNN: towards real-\ntime object detection with region proposal networks. In Cortes, C., Lawrence,\nN. D., Lee, D. D., Sugiyama, M. & Garnett, R. (eds.) Advances in Neural\nInformation Processing Systems 28: Annual Conference on Neural Information\nProcessing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada,\n91–99 (2015). URL https://proceedings.neurips.cc/paper/2015/hash/\n14bfa6bb14875e45bba028a21ed38046-Abstract.html.\n[7] Ren, D., Hu, Z. & Yuan, W. The study of cobb angular velocity in cervical\nspine during dynamic extension–flexion. Spine 41, E410–E415 (2016).\n[8] Cao, Y. et al. Preoperative cervical cobb angle is a risk factor for postoperative\naxial neck pain after anterior cervical discectomy and fusion with zero-profile\ninterbody. Orthopaedic surgery 14, 3225–3232 (2022).\n[9] Horng, M.-H., Kuok, C.-P., Fu, M.-J., Lin, C.-J. & Sun, Y.-N. Cobb angle\nmeasurement of spine from x-ray images using convolutional neural network.\nComputational and mathematical methods in medicine 2019, 6357171 (2019).\n[10] Silber,J.S.,Lipetz,J.S.,Hayes,V.M.&Lonner,B.S. Measurementvariability\nin the assessment of sagittal alignment of the cervical spine: a comparison of\nthe gore and cobb methods. Clinical Spine Surgery 17, 301–305 (2004).\n23\n[11] Kong,C.et al. Theratioofc2–c7cobbangletot1slopeisaneffectiveparameter\nfor the selection of posterior surgical approach for patients with multisegmental\ncervical spondylotic myelopathy. Journal of Orthopaedic Science 25, 953–959\n(2020).\n[12] Zhang, J., Buser, Z., Abedi, A., Dong, X. & Wang, J. C. Can c2-6 cobb angle\nreplace c2-7 cobb angle?: an analysis of cervical kinetic magnetic resonance\nimages and x-rays. Spine 44, 240–245 (2019).\n[13] Wang, C. et al. Deep learning model for measuring the sagittal cobb angle on\ncervical spine computed tomography. BMC Medical Imaging 23, 196 (2023).\n[14] Liu, J. et al. The measurement of cobb angle based on spine x-ray images using\nmulti-scale convolutional neural network. Physical and Engineering Sciences in\nMedicine 44, 809–821 (2021).\n[15] Ganaie, M.A., Hu, M., Malik, A.K., Tanveer, M.&Suganthan, P.N. Ensemble\ndeeplearning: Areview. Engineering Applications of Artificial Intelligence 115,\n105151 (2022).\n[16] Tan, L. A., Riew, K. D. & Traynelis, V. C. Cervical spine deformityâpart 1:\nbiomechanics, radiographic parameters, and classification. Neurosurgery 81,\n197–203 (2017).\n[17] Azimi, P., Yazdanian, T., Benzel, E. C., Hai, Y. & Montazeri, A. Sagittal\nbalance of the cervical spine: A systematic review and meta-analysis. European\nSpine Journal 30, 1411–1439 (2021).\n[18] Li, X.-Y., Wang, Y., Zhu, W.-G., Kong, C. & Lu, S.-B. Impact of cervical and\nglobal spine sagittal alignment on cervical curvature changes after posterior\ncervical laminoplasty. Journal of Orthopaedic Surgery and Research 17, 521\n(2022).\n[19] Aebli, N. et al. The torg-pavlov ratio for the prediction of acute spinal cord\ninjury after a minor trauma to the cervical spine. The Spine Journal 13, 605–\n612 (2013).\n[20] Nouri, A. et al. Congenital cervical spine stenosis in a multicenter global cohort\nofpatientswithdegenerativecervicalmyelopathy: anambispectivereportbased\non a magnetic resonance imaging diagnostic criterion. Neurosurgery 83, 521–\n528 (2018).\n24\n[21] Moon, M.-S., Choi, W. R., Lim, H. G., Lee, S.-Y. & Wi, S. M. Pavlov’s\nratio of the cervical spine in a korean population: a comparative study by\nage in patients with minor trauma without neurologic symptoms. Clinics in\nOrthopedic Surgery 13, 71 (2021).\n[22] Lindenmann, S., Tsagkaris, C., Farshad, M. & Widmer, J. Kinematics of the\ncervical spine under healthy and degenerative conditions: a systematic review.\nAnnals of Biomedical Engineering 50, 1705–1733 (2022).\n[23] Woiciechowsky, C., Thomale, U. W. & Kroppenstedt, S. N. Degenerative\nspondylolisthesis of the cervical spineâsymptoms and surgical strategies de-\npending on disease progress. European Spine Journal 13, 680–684 (2004).\n[24] Burke, C. J. et al. Clinical feasibility of 2d dynamic sagittal haste flexion-\nextension imaging of the cervical spine for the assessment of spondylolisthesis\nand cervical cord impingement. European Journal of Radiology 134, 109447\n(2021).\n[25] He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recog-\nnition. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, 770–778 (2016).\n[26] Lin, T.-Y. et al. Feature pyramid networks for object detection. In Proceedings\nof the IEEE conference on computer vision and pattern recognition, 2117–2125\n(2017).\n[27] Yu, Z. et al. Magnetic resonance imaging and dynamic x-rayâs correlations\nwith dynamic electrophysiological findings in cervical spondylotic myelopathy:\na retrospective cohort study. BMC Neurology 20, 367 (2020).\n[28] Zhao, Y., Sun, Y., Chen, Z.-Q. & Liu, Z.-j. Application of cervical arthroplasty\nwith bryan cervical disc: Long-term x-ray and magnetic resonance imaging\nfollow-up results. Chinese medical journal 123, 2999–3002 (2010).\n[29] Bostelmann, R. et al. Biochemical validity of imaging techniques (x-ray, mri,\nand dgemric) in degenerative disc disease of the human cervical spineâan in\nvivo study. The Spine Journal 17, 196–202 (2017).\n[30] Koeppen, D., Stelling, H., Goll, M., Kroppenstedt, S. & Cabraja, M. Com-\nparison of sagittal vertical axis and decompression on the clinical outcome of\ncervical spondylotic myelopathy. Clinical Neurology and Neurosurgery 213,\n107125 (2022).\n[31] Hofler, R. C. et al. Neutral cervical sagittal vertical axis and cervical lordosis\nvary with t1 tilt. Journal of Neurosurgery: Spine 33, 158–164 (2020).\n26"}, {"round": 2, "reviewer_comments": {"reviewer_1": "Results section\nPage 5: Are ‘patients’ and ‘cases’ used interchangeably? This might be confusing because one patient usually has multiple\ncases (multiple radiological examinations). Please be consistent in your text.\nPage 5: When you say ‘Among these, the results from MRI stenosis, MRI intervertebral discs, and X-rays were consistent.’\nwhen describing training, validation, and test sets, what do you mean by ‘consistent results’? I would not compare the\nperformance of the 3 sets because the training set has usually high performance. If you refer to consistent characteristics\n(consistent data distributions) please use the appropriate word.\nPage 7: you started a sentence with And which is not appropriate: ‘And the sub-figure Fig. 2 c shows the bias distribution between our predicted results and the gold standard\n(of Cobb angle) with Bland-Altman analysis,’\nFigures: Fig 2-b: are the violin and box plots representing the same thing? If yes, I would remove one of the 2 since they give\nalmost the same information. If you report the boxplot I would remove from the legend what I would consider common\nknowledge (IQR range, median…). Same for Fig3-b\nFig 4: remove from the legend all the boxplot element labels. It is common knowledge and people know what a boxplot\nrepresents. Maybe, if you want to keep the mean in the plot, you could just keep it in the legend. Minor comment: in this case,\nyou don’t need the colours for the boxes because it is redundant information with the x-axis ticks (you know from the ticks\nwhat each box is, you don’t need colours).\nIn Fig 5 you have the legend only in one plot. Again, the colours Fig 5 (a to d) are not needed (however, you can keep them\nif you like).\nMethods\nGeneral comment:\nI see your point in the answer to my previous review about the multi-modal model. However, I am still convinced that you\ncan’t call your model multi-modal. In deep learning multi-modal has a very specific meaning. In your case, you use two\ndifferent data modalities to evaluate different aspects of the patient and, while this is very important from a clinical point of\nview, it is not multi-modal learning.\nI also see your point on the multi-task. However, the multiple metrics are computed from the keypoints using formulae, right?\nThis means that you are not adding new neural network layers that you want to optimise. Did you mean that you optimise\nonly on the points you needed to compute the metrics? Also, the figure you provided in the response might be misleading\nbecause it seems that you computed the value of the parameter (Cobb angle for example) and you computed the loss on this\nparameter. The problem is how do you backpropagate this? I don’t think that the formula to compute the Cobb angle is\ndifferentiable with respect to the model’s parameters. Please explain better.\nPage 21: the Euclidean distance is not a concept (what do you mean by concept?). Also, in the following sentence, it is\nwrong to say the formula for the MAE is… The formula you reported is not the MAE; it is the Euclidean distance. You can tell\nthat you computed the errors as the Euclidean distance between the ground truth and the predicted points. MAE could be\npotentially applied to x and y coordinates separately but I do not think it is what you want here. There is a sentence that\nseems to be incomplete: ‘Particularly for the medical images, where most targets are relatively small to the images.’\nPage 21: The authors claim to use mean absolute error (MAE) as the loss function for localisation, yet they actually report\nthe Euclidean distance formula, which is not equivalent to MAE. Furthermore, they state that the model’s predicted\nheatmaps are converted into coordinates via an argmax operation. However, because argmax is not differentiable, one cannot\ncompute loss directly on those coordinates and backpropagate gradients through them. In practice, when performing\nheatmap regression, the loss is computed on the heatmaps themselves, not on the extracted coordinates. The paper does\nnot make this distinction clear. I would ask the authors to check that.\nPage 26 Evaluation metrics: do not start the sentence with And: ‘And the performance\nmetrics are evaluated through accuracy, precision, sensitivity, specificity and F1 score.’\nDiscussion\nI would not start the discussion with ‘Therefore’\nGeneral comment\nThere is no description of the dataset in terms of image resolution and MRI scan acquisition. In deep learning, an important\nrole is played by image preprocessing. Did you use any data augmentation techniques? You did not provide any detail on\nthe model: batch size, learning rate, weight decay, whether you used the pre-trained weights or you trained from scratch.\nI also reviewed Reviewer 2’s report and can confirm that all comments except Q3 have been addressed. In Q3 the authors\ndid not reply to the issue about overfitting that was raised by the reviewer."}, "author_response": "Response to the Reviewers’ Comments\nWe appreciate the thorough evaluation of our study by the reviewers and editors.\nIn the revised version of our manuscript, we address essentially all of the points\nraised by the reviewers, and we think the resulting study is much improved. Specific\nrevisions and responses to each comment are provided in detail below.\nPreliminary notes:\n• Sectionandpagerefertothedouble-columnversionoftherevisedmanuscript.\n• We use two complementary conventions to refer to tables (or figures): Table\n1 refers to TABLE 1 of the double column version of the revised manuscript,\nwhile mark like Table R1 refers to this response letter.\n• References are re-organized (including the reference numbers in the com-\nments of reviewer) in this response paper, all of them are listed at the end of\nthe response.\n• Better review in color. The texts in red, black and blue are the comments\nof reviewers, our response and revised text (in double column manuscript),\nrespectively.\nReviewer #1\nResults section\nQ1. Page 5: Are ‘patients’ and ‘cases’ used interchangeably? This might be confusing\nbecause one patient usually has multiple cases (multiple radiological examinations).\nPlease be consistent in your text.\nResponse 1. Thanks for your comments. ‘Patients’ and ‘cases’ were equivalent in\nour experimental setup, with each patient corresponding to a single case. To avoid\nconfusion, we will uniformly use ‘patients’ throughout the revised manuscript.\nintervertebral discs, and X-rays were consistent.’ when describing training, valida-\ntion, and test sets, what do you mean by ‘consistent results’? I would not compare\nthe performance of the 3 sets because the training set has usually high performance.\nIf you refer to consistent characteristics (consistent data distributions) please use\nthe appropriate word.\nResponse2. Weapologizefortheambiguityhere. ‘Amongthese, theresultsfrom\nMRI stenosis, MRI intervertebral discs, and X-rays were consistent’ means for the\nvast majority of cases, the three annotators’ results were consistent (The annotation\nresults reached consensus). In cases of disagreement, the annotations from a more\nauthoritative expert shall prevail. We have revised the manuscript as follows:\n‘Forthevastmajorityofcases,thethreeannotators’resultsarelargelyconsistent.\nIn cases of disagreement, the annotations from a more authoritative expert shall\nprevail.’\nQ3. Page 7: you started a sentence with And which is not appropriate: ‘And the\nsub-figure Fig. 2 c shows the bias distribution between our predicted results and the\ngold standard (of Cobb angle) with Bland-Altman analysis,’\nResponse 3. Thanks for your useful comments. We will remove the ‘And’ at the\nbeginning of the sentence to enhance the formality of the expression, and we will\nalso check other related instances in the manuscript to ensure academic rigor. The\nrevised version is as follows:\n‘The sub-figure Fig. 2 c shows the bias distribution between our predicted results\nand the gold standard (of Cobb angle) with Bland-Altman analysis, and the 95%\nlimits of agreement (± 1.96SD) were [-6.49◦, 6.31◦].’\nQ4. Figures: Fig 2-b: are the violin and box plots representing the same thing?\nIf yes, I would remove one of the 2 since they give almost the same information. If\nyou report the boxplot I would remove from the legend what I would consider common\nknowledge (IQR range, median...). Same for Fig3-b.\nResponse 4. Thanks for your insightful comments. Since violin plots and box\nplots convey similar dataset information, we retained only the violin plots, which\ncomprehensively display all data points, while removing the box plots. This adjust-\nment also resolves the issue of redundant information in the legend of box plots.\nThis modification has been applied to both Fig 2-b and Fig 3-b (see Fig. R1 and\nFig. R2).\nQ5. Fig 4: remove from the legend all the boxplot element labels. It is common\nknowledge and people know what a boxplot represents. Maybe, if you want to keep the\nmean in the plot, you could just keep it in the legend. Minor comment: in this case,\nC 3\nC 4\nC 5\nC 6\nC 7\na c\ne\nAverage (°)\nb d\nFigure R1: The revised Fig. 2-b.\na c\ne\nAverage (mm)\nb d\nFigure R2: The revised Fig. 3-b.\n3\na 1 Spondylolisthesis angl b c Spondylolisthesis angle/distance accuracy\nSpondylolisthesis angle Spondylolisthesis distance\nC2-C3\n100%\n80%\n60%\n40%\nC6-C7 C3-C4\n20%\n0%\nC5-C6 C4-C5\nFigure R3: The revised Fig. 4.\nyou don’t need the colours for the boxes because it is redundant information with the\nx-axis ticks (you know from the ticks what each box is, you don’t need colours).\nResponse 5. Thanks for your valuable comments. We have removed all the\nboxplotelementlabelsfromthelegendtoachievemoreconcisevisualrepresentations\nwithout redundant information. Additionally, we simplified the graphs by omitting\ncolors, since x-axis ticks adequately convey the essential information (see Fig. R3).\nQ6. In Fig 5 you have the legend only in one plot. Again, the colours Fig 5 (a\nto d) are not needed (however, you can keep them if you like).\nResponse 6. Thanks for your useful comments. We have uniformly removed all\nboxplot element labels and eliminated color differentiation (see Fig. R4).\nMethods\nGeneral comment:\nQ7. I see your point in the answer to my previous review about the multi-modal\nmodel. However, I am still convinced that you can’t call your model multi-modal. In\ndeep learning multi-modal has a very specific meaning. In your case, you use two\ndifferent data modalities to evaluate different aspects of the patient and, while this\nis very important from a clinical point of view, it is not multi-modal learning.\nResponse 7. Thanks for your thorough review and valuable comments. We agree\nthat using two different data modalities to diagnose patients is more important from\na clinical point of view. We also agree that our proposed method represents a multi-\nmodal collaboration framework rather than multi-modal learning. Our multi-modal\ncollaboration framework generates complementary clinical indicators from diverse\nperspectives, enabling clinical doctors to formulate comprehensive and precise op-\nerative plans through synthesized diagnostic information.\nWe also agree that the classical multi-modal learning paradigm indeed requires\nthe joint training of one model (or a combined model) with multi-modal data to\nachieve feature fusion and unified prediction. However, our approach is a multi-\nmodal collaboration framework, in which multiple models receive inputs of multi-\n4\na b\nc d PavloPvav lroav triaoti oa accccuurarcay cy\n80.0%\n76.0%\n74.5%\n75.0% 73.5%\n69.6%\n70.0%\n65.0%\n60.0%\n55.0%\n50.0%\nC3-C5 C4-C6 C5-C7 overall\nFigure R4: The revised Fig. 5.\n5\nmodal data, and the predicted results are jointly used to solve a complex task. The\nintegration of multi-modal information occurs at the decision-making level, which is\ndistinct from multi-modal learning that operates at the feature level. Specifically,\nour model receives inputs from two modalities, including X-ray and MR images,\nindependently predicts multiple clinical indicators from each modality, and integrates\nall indicators to generate comprehensive diagnostic conclusions.\nFrom the perspective of clinical doctors’ needs, collaboration is more reasonable\nand better aligned with real-world clinical practice. Clinical doctors typically lever-\nage different imaging modalities for distinct clinical purposes based on their unique\ncharacteristics, rather than fusing them for a unified diagnostic goal. For instance,\ncost-effective X-rays are primarily employed for initial evaluations such as assessing\nspinal curvature and alignment, while higher-cost MRI with superior imaging qual-\nity enables more precise identification of compressive or stenotic lesions. Moreover,\nclinical doctors employ these multi-modal data in a sequential workflow, typically\nacquiring X-rays first followed by MRI, which fundamentally differs from the con-\nventional multi-modal learning paradigm that requires simultaneous fusion of all\ninput data.\nWe will clearly specify our multi-modal collaboration framework in the revised\nversion as follows:\n‘ In this work, we propose a multi-modal collaboration cervical spondylosis di-\nagnosis framework trained with multi-task learning, oriented to the clinical require-\nments of less annotation workload for doctors, and fine-grained measurements of\nmultiple types of degeneration.\nThe multi-modal diagnostic results collaborate complementarily to support the\ndoctors’ needs, reflecting real-world clinical practice where doctors integrate find-\nings from different imaging modalities to address distinct aspects of the diagnostic\nprocess.’\nQ8. I also see your point on the multi-task. However, the multiple metrics are\ncomputed from the keypoints using formulae, right? This means that you are not\nadding new neural network layers that you want to optimise. Did you mean that\nyou optimise only on the points you needed to compute the metrics? Also, the figure\nyou provided in the response might be misleading because it seems that you computed\nthe value of the parameter (Cobb angle for example) and you computed the loss on\nthis parameter. The problem is how do you backpropagate this? I don’t think that\nthe formula to compute the Cobb angle is differentiable with respect to the model’s\nparameters. Please explain better.\nResponse 8. Thanks for your comments. We indeed calculate the indicators\nFeaturemap\nloss\nLocalization Coordinates\nFasterR-CNN:\nloss\nClassification Labels\nCNN\nKey\npoints\ncoordinates\na\nloss\nb\nCobb angle Labels\nloss\nSVA value Labels\nOurs:\nloss\nSpondylolisthesis\nLabels\nangle\n…\nFigure R5: Multi-task setting of the Faster R-CNN and our method. (a)Multi-task\nlearning framework of the Faster R-CNN. (b)Multi-task learning framework of our\nproposed method.\nusing geometric formulations based on keypoints coordinates, without adding neural\nnetwork layers. Furthermore, as illustrated in the figure provided in our previous\nround response (see Fig. R5), we compute the loss between the predicted and the\nground truth values for all indicators and backpropagate this loss to the keypoints\ndetection network. We employed the Differentiable Spatial to Numerical Transform\n(DSNT) to convert heatmaps into coordinate representations, so that the entire\ncomputationalprocessisdifferentiablewithrespecttothemodel’sparameters. More\ndetailed computational procedures will be explained and updated in the revised\nmanuscript.\n‘ In our multi-task learning framework, we employ the Differentiable Spatial to\nNumerical Transform (DSNT) [1] to convert heatmaps into coordinate represen-\ntations through a differentiable spatial expectation operation, enabling end-to-end\nbackpropagationoferrorsfrommultipleclinicalindicatorstothekeypointsdetection\nmodule. Given the predicted heatmap H ∈ Rh×w, we apply the softmax function\nfor normalization to ensure the results conform to a probability distribution:\nexp(H )\nH ˆ = i,j (1)\ni,j Σh Σw exp(H )\nk=1 l=1 k,l\nWe then construct two h × w matrices X and Y, where X = 2j−(w+1)\nij w\nij h\neach element contains the x- (for X) or y-coordinate (for Y) respectively, such\nthat the top-left corner of the image is at (-1,1) and bottom-right is at (1,1). The\npredicted coordinates (xˆ,yˆ) are derived through Frobenius inner product between\nthe normalized heatmap Z ˆ and coordinate matrices X,Y.\n(xˆ,yˆ) = DSNT(Z ˆ ) = [⟨Z ˆ ,X⟩ ⟨Z ˆ ,X⟩ ] (2)\nF F\nwhere ⟨·,·⟩ denotes the Frobenius inner product, which is equivalent to the\nF\nscalar dot product of vectorized matrices. Essentially, this operation computes a\nweighted average of the coordinate grid with respect to the heatmap’s probability\ndistribution, yielding the mathematical expectation coordinates under this distribu-\ntion. Sinceboththesoftmaxandexpectationoperationsaredifferentiable, gradients\ncan be backpropagated seamlessly.’\nFor each keypoint, we predict a corresponding heatmap, resulting in the coordi-\nnate matrix P′ as follows:\n \nx y x y x y x y\n1 1 2 2 3 3 4 4\n. . . . . . . .\nP′ =   . . . . . . . . . . . . . . . .   (3)\n \nx y x y x y x y\n21 21 22 22 23 23 24 24\n6×8\nwherepointsp representtheantero-superior,antero-inferior,postero-superior,\n1to4\nand postero-inferior corner points of the C2 vertebrae, as well as so on. Using the\nCobb angle measurement procedure as an example, the angle points of the lower\nedges of the C2 and C7 vertebrae are used to calculate the Cobb angle. The de-\ntailed computational formula is presented below:\np = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) (4)\n2x [0,2] 2y [0,3] 4x [0,6] 4y [0,7]\np = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) ,p = (cid:0) P′ (cid:1) (5)\n22x [5,2] 22y [5,3] 24x [5,6] 24y [5,7]\nk1 = (p −p )/(p −p +δ), k2 = (p −p )/(p −p +δ), δ = 1e−9\n2y 4y 2x 4x 22y 24y 22x 24x\n(6)\nQ = arctan((k1−k2)/(1−k1×k2))×(180◦/π) (7)\nCA\n2 4\nlower edge, while k1 denotes its slope. Correspondingly, in Eq. 5 and Eq. 6, p\n22\nand p indicate the two vertices of the C7 lower edge, while k2 denotes its slope.\nlines in Eq. 7. Since Cobb angle computation involves only linear operations and\ntrigonometric calculations, the loss between predicted and ground truth values can\nbe backpropagated to the model’s parameters. The computation processes for other\nindicators are also presented in the manuscript, with all involved operations being\ndifferentiable.\nOur framework performs end-to-end optimization of all detected keypoints, re-\ngardless of whether they are used for clinical indicators computation. The overall\nloss function L is composed of two terms: loss L for the DSNT-based\nOurs multi−task\nmetric measurement task and loss L′ for the heatmap keypoint detection task,\nheatmap\nrepresented as:\nL = L +L′ (8)\nOurs multi−task heatmap\nTherefore, even for keypoints not involved in indicator calculations, optimization\nis still achieved through the heatmap loss L′ .\nheatmap\nQ9. Page 21: the Euclidean distance is not a concept (what do you mean by\nconcept?). Also, in the following sentence, it is wrong to say the formula for the\nMAE is... The formula you reported is not the MAE; it is the Euclidean distance.\nYou can tell that you computed the errors as the Euclidean distance between the\nground truth and the predicted points. MAE could be potentially applied to x and\ny coordinates separately but I do not think it is what you want here. There is a\nsentence that seems to be incomplete: ‘Particularly for the medical images, where\nmost targets are relatively small to the images.’\nResponse 9. Thanks for your correction. We employ Euclidean distance as the\nquantitative metric for error measurement. Although the provided formula correctly\ncalculates the Euclidean distance between the ground truth and the predicted points\nas the error term, it was erroneously labeled as ‘MAE’ (Mean Absolute Error). We\nwill correct all instances of ‘MAE’ to ‘MSE’ (Mean Squared Error) in the revised\nmanuscript, as MSE is mathematically equivalent to Euclidean distance in this con-\ntext.\nThe loss for heatmap supervision is computed using the Mean Squared Error\n(MSE) between the ground-truth heatmap H and the predicted one H ¯, which is\nmathematically equivalent to the Euclidean distance. The calculation is performed\nas follows:\n1 (cid:88)\nMSE(H,H ¯ ) = (x −x¯)2 (9)\ni i\nN\ni=1\nwhere N is the number of points in the heatmap, x and x¯ represent the ground-\ntruth probability and the predicted probability of the i-th pixel.\nWe will also revise the incomplete sentence identified by the reviewer to ensure\na clear expression of the intended meaning:\n‘Heatmapsaretypicallygeneratedatlowerresolutionsthroughdown-samplingto\nbalance computational efficiency and receptive field coverage, leading to localization\nerrors. Particularly for the medical images, where most targets are relatively small\nto images, minimizing localization errors becomes critical. Our keypoint detection\nmodule is designed to generate the heatmap on the highest resolution feature map,\n...’\nQ10. Page 21: The authors claim to use mean absolute error (MAE) as the loss\nfunction for localization, yet they actually report the Euclidean distance formula,\nwhich is not equivalent to MAE. Furthermore, they state that the model’s predicted\nheatmaps are converted into coordinates via an argmax operation. However, because\nargmax is not differentiable, one cannot compute loss directly on those coordinates\nand backpropagate gradients through them. In practice, when performing heatmap\nregression, the loss is computed on the heatmaps themselves, not on the extracted\ncoordinates. The paper does not make this distinction clear. I would ask the authors\nto check that.\nResponse 10. Thanks for your comments. We would like to clarify that we em-\nployed MSE (Mean Squared Error) as the loss function, as detailed in Response 9,\nand this correction will be incorporated into the revised manuscript. Furthermore,\nwe employ the Differentiable Spatial to Numerical Transform (DSNT) rather than\nthe argmax operation for converting heatmap to coordinates, a methodological dis-\ntinction we will emphasize in the revised manuscript. Regarding the loss function\ndesign, we simultaneously compute losses on both the heatmap and clinical indi-\ncators, as specified in Eq. 8. The adoption of DSNT enables the backpropagation\nof clinical indicator losses through the keypoint detection network. The computa-\ntional details are explained in Response 8 and will be emphasized in our revised\nmanuscript.\nQ11. Page 26 Evaluation metrics: do not start the sentence with And: ‘And the\nperformance metrics are evaluated through accuracy, precision, sensitivity, specificity\nand F1 score.’\nResponse 11. Thanks for your comments. We have revised this sentence as\n10\nfollows: ‘The performance metrics are evaluated through accuracy, precision, sensi-\ntivity, specificity and F1 score.’\nWe have also revised the other 11 sentences starting with ‘And’.\nDiscussion\nQ12. I would not start the discussion with ‘Therefore’\nResponse 12. Thanks for your comments. We have revised this sentence as fol-\nlows: ‘In spinal surgery, the concept of Cobb angle is used to describe the alignment\nand curvature of the spine. ’\nGeneral comment\nQ13. There is no description of the dataset in terms of image resolution and MRI\nscan acquisition. In deep learning, an important role is played by image prepro-\ncessing. Did you use any data augmentation techniques? You did not provide any\ndetail on the model: batch size, learning rate, weight decay, whether you used the\npre-trained weights or you trained from scratch.\nResponse 13. Thanks for your useful comments. The width of the original X-ray\nimages ranges from 908 to 2973 pixels, and the height ranges from 1125 to 3032\npixels. With reference to [7, 8], we resized all images uniformly to 512×512 pixels\nand performed normalization operations to standardize input distributions.\nAs for the detailed acquisition parameters for our MRI dataset, MRI cervical\nspine studies were scanned using different MRI scanners (GE 1.5- and 3.0-T plat-\nforms, Siemens 3.0 T platforms, United Imaging 3.0 T platforms) with the same\nsequences and standard phased-array surface coils. Axial T2-weighted Digital Imag-\ning and Communications in Medicine images were used. All studies were performed\nin the supine position with head-neck combined coil / CTL coil. Table. R1 provides\ndetails on the MRI scanners and sequences.\nRegarding the hyperparameter configuration for model training, we set the batch\nsize to 32. The model was initialized with the pre-trained HigherHRNet [2] weights\nprovided by MMPose [3], and trained for 300 epochs using the Adam optimizer with\na fixed learning rate of 1.5×10−3. The training configuration will be incorporated\ninto the revised version of the manuscript.\nQ14. I also reviewed Reviewer 2’s report and can confirm that all comments\nexcept Q3 have been addressed. In Q3 the authors did not reply to the issue about\noverfitting that was raised by the reviewer.\nThe Q3 of Reviewer 2 in the last round was as follows: ‘ Q3. The manuscript\n= echo time, TR = repetition time, FOV = Field of view, GE = General Electric.\nMRI Sequences FOV(mm2) Matrix Slice Gap(mm) TR(ms) Flip\nscanners thick- / angle(◦)\nness(mm) TE(ms)\nGE 1.5 T MERGE 200×150 272×192 4 0.8 433.1/5.5 20\nGE 3.0 T MERGE 256×128 272×192 4 0.8 552.9/12.0 20\n(HDXT)\nSiemens MEDIC 190×140 320×256 4 0.4 400.0/17.0 20\n3.0 T\n(Prisma)\nUnited Imaging 3.0 T\nuMR780 GETI 200×150 272×245 4 0.4 592.1/13.45 20\nuMR880 GETI 200×150 272×245 4 0.4 608/13.33 20\nsuggests a particular sensitivity of the model to certain indicators, such as the SVA\nvalue and spondylolisthesis distance. However, there is a lack of discussion regard-\ning the model’s performance variability across different conditions and its potential\nvulnerability to overfitting on these indicators. Addressing overfitting is crucial to\nensure the model’s applicability and effectiveness in diverse real-world scenarios,\nwhere patient presentations can vary widely.’\nResponse 14. Thanks for your comments. Our proposed methodology effec-\ntively mitigates overfitting and demonstrates strong generalizability across diverse\nreal-world scenarios. We agree that our models are more sensitive to distance and\nposition based indicators, such as SVA value and spondylolisthesis distance rather\nthan angle based indicators, such as Cobb angle and spondylolisthesis angle. While\nsome work [5, 6] considered a single criterion as ‘20% of displacement’, our work\nimplemented the composite criterion [4], which incorporates both distance measure-\nment and angle measurement. This approach effectively prevents model overfitting\non the distance and position based indicators, thereby significantly enhancing model\nrobustness.\nAnalysis of our dataset distribution demonstrates robust model generalizability\nacross diverse real-world clinical scenarios, effectively mitigating overfitting con-\ncerns. This generalizability is evidenced by the following aspects:\n1. Multi-center design. Patients were recruited from two distinct hospitals\n(Peking University Third Hospital and Beijing Haidian Hospital), ensuring in-\nstitutionaldiversityanddemonstratingtheapplicabilityacrossdiversehospital\nsettings.\n12\n2. Comprehensive clinical coverage. Participants were drawn from both\noutpatient and inpatient settings, reflecting the full spectrum of real-world\nclinical scenarios.\n3. Demographic diversity. The study cohort demonstrated balanced demo-\ngraphic representation, with a mean age of 54 years (SD = 10.28, range of 21-\n84 years) and the gender distribution of 60.6% male versus 39.4% female. This\nage-inclusive and sex-balanced population ensures the broad clinical applica-\nbility of our findings across diverse cervical spondylosis patient populations.\n4. Heterogeneous patient presentations. The cervical spondylosis patients\nin this study exhibited heterogeneous patient presentations, encompassing\nboth radiculopathy (primarily presenting with radiating pain or numbness in\nthe neck-shoulder region and upper extremities) and myelopathy (character-\nized by limb numbness and weakness), rather than being limited to a single\nsubtype. Specifically, the patient presentations in our test set (230 patients)\ndemonstrated clinical heterogeneity, with a mean symptom duration of 40.12\nmonths (standard deviation = 66.67, range of 1-480 months). The presenting\nsymptoms included numbness in 179 patients (77.83%), pain in 150 patients\n(65.22%),andweaknessin123patients(53.48%),sinceindividualpatientsmay\npresent with single or multiple concurrent symptoms. Meanwhile, 222 patients\n(96.52%) had symptoms distributed in the limbs and 129 patients (56.09%)\nhad symptoms distributed in the neck and shoulders, given that patients may\npresentwithsingleormultiplesymptomaticregions. Basedonthesestatistical\ndata, the patient presentations in our study cohort demonstrate substantial\nheterogeneity, ensuring broad applicability across diverse real-world clinical\nscenarios.\n[1] Nibali, A., He, Z., Morgan, S.&Prendergast, L. Numericalcoordinateregression\nwith convolutional neural networks. arXiv preprint arXiv:1801.07372 (2018).\n[2] Cheng, B., Xiao, B., Wang, J. et al. Higherhrnet: Scale-aware representation\nlearning for bottom-up human pose estimation (2020). URL https://doi.org/\n10.1109/CVPR42600.2020.00543.\n[3] Contributors, M. Openmmlab pose estimation toolbox and benchmark. https:\n//github.com/open-mmlab/mmpose (2020).\n[4] Maeda, T. et al. Soft-tissue damage and segmental instability in adult patients\nwith cervical spinal cord injury without major bone injury. Spine 37, E1560–\n1566 (2012). URL https://api.semanticscholar.org/CorpusID:10286201.\n13\n[5] Xiao, Q. et al. Application of tvd-net for sagittal alignment and instability\nmeasurements in cervical spine radiographs. Medical Physics 50, 4182–4196\n(2023).\n[6] Girard, V. et al. Post-traumatic lower cervical spine instability: Arthrodesis\nclinical and radiological outcomes at 5years. Orthopaedics & Traumatology:\nSurgery & Research 100, 385–388 (2014). URL https://www.sciencedirect.\ncom/science/article/pii/S1877056814000814.\n[7] Tanno, R. et al. Collaboration between clinicians and vision–language models in\nradiology report generation. Nature Medicine 31, 599–608 (2025).\n[8] Niu, C. et al. Medical multimodal multitask foundation model for lung cancer\nscreening. Nature Communications 16, 1523 (2025).\n14"}, {"round": 3, "reviewer_comments": {"reviewer_1": "I would like to thank the authors for their work. I still have a few minor points on the Methods section.\nIn the Detection paragraph after a few lines: Are you talking about your previous model? If no, I would use the generic 'A\nprevious detection model...'. Especially because it is the first time this model is mentioned. Also when you say 'the similar\npipeline...'. Also che English spell of the following sentence.\nIn the following paragraph ('The ground truth pixel position is usually converted...'): Are you talking about your work or in\ngeneral? This 'usually' sounds like you are talking in general.\nWhen you say MSE is equivalent to Euclidean: That's not true. Euclidean distance is a distance between 2 points. MSE is\nan error squared between 2 values in you case (the probability at a specific location). If you computed the MSE pixel-wise\n(pixel in position i,j in predicted heatmap and in position i,j in the ground truth heatmap) it is OK but you can't say MSE =\nEuclidean distance. Also the formula is confusing because you are using x which to me is the x-coordinate only. I would\nchange maybe using lower case h. Please be consistent with the heatmaps notation: you are using lower case h, upper\ncase H, and H hat. \nDetection vs Degeneration Diagnosis in Multi Task Learning paragraphs: Is it correct if I understood that for detection you\nused a hm loss only while now you are computing the coordinates using the DSNT layer and then computing the loss on the\npredicted coordinates? Why a different approach now?\n(Remarks on code availability)\nThe code provided is not useful to reproduce results. I believe it is also incomplete because a lot of modules presented were\nmissing. The authors could have provided some fake images to be able to at least run the code. Only the binary classifier of\nintervertebral disc degeneration is available."}, "author_response": "Response to the Reviewers’ Comments\nWe appreciate the thorough evaluation of our study by the reviewers and editors.\nIn the revised version of our manuscript, we address essentially all of the points\nraised by the reviewers, and we think the resulting study is much improved. Specific\nrevisions and responses to each comment are provided in detail below.\nPreliminary notes:\n• Sectionandpagerefertothedouble-columnversionoftherevisedmanuscript.\n• We use two complementary conventions to refer to tables (or figures): Table\n1 refers to TABLE 1 of the double column version of the revised manuscript,\nwhile mark like Table R1 refers to this response letter.\n• References are re-organized (including the reference numbers in the com-\nments of reviewer) in this response paper, all of them are listed at the end of\nthe response.\n• Better review in color. The texts in red, black and blue are the comments\nof reviewers, our response and revised text (in double column manuscript),\nrespectively.\nReviewer #1\nI would like to thank the authors for their work. I still have a few minor points on\nthe Methods section.\nQ1. In the Detection paragraph after a few lines: Are you talking about your\nprevious model? If no, I would use the generic ‘A previous detection model...’.\nEspecially because it is the first time this model is mentioned. Also when you say\n‘the similar pipeline...’. Also check English spell of the following sentence.\nResponse 1. Thanks for your comments. We are not referring to our previous\nmodel, but rather the keypoint detection models in other mainstream works. We\n1\nhave revised the expression here to ‘Previous keypoint detection models usually\nfollow a similar pipeline to...’ to eliminate ambiguity. Also, we have checked the\nEnglish spelling of other sentences.\nQ2. In the following paragraph (‘The ground truth pixel position is usually con-\nverted...’): Are you talking about your work or in general? This ‘usually’ sounds\nlike you are talking in general.\nResponse 2. Thanks for your comments. As the reviewers pointed out, this\npart is describing the general practice in the field [1, 2, 3], and our work shares a\nsimilar approach. Like common practices, we also convert the ground truth into\nheatmaps to calculate the error. However, unlike most methods that compute the\nloss on down-sampled low-resolution heatmaps, we calculate the loss on the highest\nresolution heatmap to minimize localization error.\nQ3. When you say MSE is equivalent to Euclidean: That’s not true. Euclidean\ndistance is a distance between 2 points. MSE is an error squared between 2 values in\nyou case (the probability at a specific location). If you computed the MSE pixel-wise\n(pixel in position i,j in predicted heatmap and in position i,j in the ground truth\nheatmap) it is OK but you can’t say MSE = Euclidean distance. Also the formula\nis confusing because you are using x which to me is the x-coordinate only. I would\nchange maybe using lower case h. Please be consistent with the heatmaps notation:\nyou are using lower case h, upper case H, and H hat. \nResponse 3. Thanks for your comments. We agree that there is a distinction\nbetween Euclidean distance and MSE. We use MSE to compute the loss, and we\nhave removed the expression ‘equivalent to the Euclidean distance’. Our loss indeed\ncalculates the error between the ground truth probability and the predicted proba-\nbilityateachpixelontheheatmap, ratherthanthedistancebetweenpixelpositions.\nMeanwhile, we have unified the notation for heatmaps. To be consistent with the\nheatmaps notation, we use upper case H to denote the ground truth heatmap, H\nhat (‘H ˆ’) to represent the predicted heatmap, and lower case h to indicate the value\nat a specific pixel location in the heatmap. The revised version is as follows:\nThe ground truth pixel position is usually converted into a heatmap, denoted as\nH , which is generated by a 2D Gaussian function with a standard deviation of 1.\nk\nThe loss for heatmap supervision is computed using the Mean Squared Error (MSE)\nbetween the ground-truth heatmap H and the predicted one H ˆ. The calculation is\nperformed as follows:\nN\n1 (cid:88)\nMSE(H,H ˆ ) = (h −h ˆ )2 (1)\ni i\nN\ni=1\n2\nwhere N is the number of points in the heatmap, h and h ˆ represent the ground-\ni i\ntruth probability and the predicted probability of the i-th pixel.\nThe loss function of heatmap supervision L can be formulated as:\nL = L = MSE(H,H ˆ ) (2)\nheatmap\nQ4. Detection vs Degeneration Diagnosis in Multi Task Learning paragraphs: Is\nit correct if I understood that for detection you used a hm loss only while now you\nare computing the coordinates using the DSNT layer and then computing the loss on\nthe predicted coordinates? Why a different approach now?\nResponse 4. Thanks for your comments. There might be some subtle differences\nbetween this and the reviewer’s understanding. We incorporate both the heatmap\nloss and the predicted coordinate loss, rather than replacing the loss function, as\nshown in the following equation (Eq.(6) in the manuscript).\n(cid:16) (cid:17) (cid:16) (cid:17)\nL = L +L′ = (cid:80)M λ MSE Q ,Q ˆ + (cid:80)N MSE H ,H ˆ\nOurs multi−task heatmap m=1 m m m n=1 nk nk\n(3)\nBy using both loss functions simultaneously, we can optimize both the accuracy\nof keypoint detection and the predictive accuracy of degenerative indicators.\nQ5. The code provided is not useful to reproduce results. I believe it is also\nincomplete because a lot of modules presented were missing. The authors could have\nprovided some fake images to be able to at least run the code. Only the binary\nclassifier of intervertebral disc degeneration is available.\nResponse 5. Thanks for the comments. We have updated the associated code\nrepository to ensure full reproducibility. The repository is available at: https:\n//github.com/isiay/Diagnostic-analysis-of-multimodal-cervical-degen\nerative-diseases.\n[1] Wang, J., Sun, K., Cheng, T. et al. Deep high-resolution representation learn-\ning for visual recognition. IEEE TRANSACTIONS ON PATTERN ANALY-\nSIS AND MACHINE INTELLIGENCE 43, 3349–3364 (2021 OCT 1). URL\nhttps://doi.org/10.1109/TPAMI.2020.2983686.\n[2] Cheng, B., Xiao, B., Wang, J. et al. Higherhrnet: Scale-aware representation\nlearning for bottom-up human pose estimation (2020). URL https://doi.or\ng/10.1109/CVPR42600.2020.00543.\n3\n[3] Sun, K., Xiao, B., Liu, D. et al. Deep high-resolution representation learning for\nhuman pose estimation (2019). URL https://doi.org/10.1109/CVPR.2019.\n00584. Proceedings Paper.\n4"}]

Decision letter