Predicting 3D forearm fracture angle from biplanar Xray images with rotational bone pose estimation

Authors: hanxue  gu
roy  colglazier
jikai  zhang
robert  lark
benjamin  alman
maciej  mazurowski
r  gu
j  colglazier
r l zhang
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 20:33:45

Abstract: Two-dimensional X-ray images, while widely used, have limitations to reflect 3D information of the imaged objects. 
Several studies have tried to recover such information from multiple X-ray images of the same object. Still, those approaches often fail due to the unrealistic assumption that the target does not move between views and those two views are perfectly orthogonal.
A problem where 3D information would be highly valuable but is very difficult to assess from 2D X-ray images is the measurement of the actual 3D fracture angles in the forearm. 
To address this problem, we propose a deep learning-based method that predicts the rotational movement and skeletal posture from biplanar X-ray images, offering a novel and precise solution.
Our strategy comprises the following steps: (1) automatic segmentation of the ulna and radius bones of the forearm on two X-ray images by a neural network; (2) prediction of the rotational parameters of the bones by a pose prediction network; (3) automatic detection of fracture locations and assessment of the fracture angles on 2D images; and (4) reconstruction of the real 3D fracture angle by inferring it from the 2D fracture information and the skeleton pose parameters collected from the two images. 
Our experiments on X-ray images show that our method can accurately measure 2D fracture angles and infer the pose of the forearm bones. By simulating X-ray images for various types of fractures, we show that our method could provide more accurate measurements of fracture angles in 3D. We are the first attempt for the fully automatic fracture angle measurements on both 2D and 3D versions, and we show the robustness of our method even in extreme cases where the two views are highly nonorthogonal.

Main paper: 
 introduction: 
 X-ray, known for its low radiation and cost, is the primary choice for diagnosing forearm fractures. However, its 2D projection nature obscures vital details, such as precise fracture angles and posture variations. Specifically, when patients undergo follow-up X-rays to Figure 1 : An illustration of a bi-planar view of X-ray images, ideal cases, and real-life clinical practice. The left shows the ideal case where the AP view and Lateral views are orthogonal to each other, while in the real case, the object changes posture across the views and will bring about the misalignment and non-perfect views.assess forearm fractures, replicating the exact arm positioning from previous images is challenging. This complexity makes it difficult for clinicians to discern whether fracture angulation changes reflect clinical progression or are mere artifacts of different positioning. Such discrepancies can hinder precise diagnosis, surgical preparation, and post-treatment evaluation by physicians (Macken et al., 2022) . Some earlier works (Fotsin et al., 2019; Lamecker et al., 2006; Ehlke et al., 2013; Chen and Fang, 2019; Kasten et al., 2020; Ying et al., 2019; Shrestha et al., 2022; Shiode et al., 2021) attempted to reconstruct 3D bones from biplanar X-ray images. These 3D reconstruction methods using the most commonly applied anteroposterior (AP) view and lateral (LAT) view, rely on the two views being entirely orthogonal to one another and the object not changing position or moving during the capture (Gu et al., 2024) , see Figure 1 (ideal case). However, in real practice, these two views refer to the anatomical position, which indicates a relatively fixed X-ray shooting angle, and the object's own moving and rotating (see Figure 1 for real cases). Obtaining two views with complete skeletal orthogonality is impossible, especially the limitation of movement caused by fractures; see Appendix A (ill-posed cases) for examples. Therefore, it is usually infeasible to reconstruct the forearm bones from two clinically taken X-ray images with the assumption of orthogonality. A relevant work (Abe et al., 2019) tried to predict the rotational movements of the healthy forearm. However, the required 2D-3D X-ray and CT pairs are difficult to obtain, and their work was limited only to pose estimation.Meanwhile, fracture assessment is an important task for X-ray-based diagnosis. Some work automatically detects fractures (Guan et al., 2020; Thian et al., 2019) or provides different types of fracture classification (Yadav and Rathor, 2020 ). Yet, accurately determining fracture angles requires more sophisticated information and has not been accomplished through a fully automated measurement process. Our algorithm not only achieves the first automatic fracture angle measurements but also considers the loss of third-dimensional information caused by X-ray images, which renders forearm posture estimation and reconstructs the fracture angle in 3D.Figure 2 : An illustration of the 3D fracture angle we measure. The 3D-coordination system applied in our work was to make the projected receptor in the yz plane. It shows that the rotation around the z-axis and y-axis would influence the 2D angle measured in the 2D X-ray images. The 2D angle varies a lot between different positions of bones, which causes inconsistent angle measurements between X-ray images and possibly inaccurate measurements compared with the real angle.In our method, we combine (1) the forearm bone segmentation algorithm to extract two forearm bones separately and (2) the pose regression network to predict the posture of bones in two randomly collected views. This process establishes a spatial relationship between the two images in 3D space. (3) a mathematical algorithm to correlate the fracture angles measured in both images with their real-world 3D counterparts. Our method is the first to attempt to achieve fully automatic 2D fracture angle measurements and to reconstruct 3D fracture angles from nonorthogonal 2D X-ray images. It is free from X-ray images and 3D image pairs or planar markers during the training stage, which differs from the previous 2D-3D reconstruction works (Shrestha et al., 2022; Shiode et al., 2021) . Our work can also be easily extended to other body parts with 2D X-ray-based fracture angle or bending angle measurements in clinical usage. 
method: 
 We detail our method for 3D fracture angle prediction in this section. Our algorithm comprised three main components: (1) measure fracture angles in 2D images separately;(2) get the positioning of bones on these two images; (3) combine this information to reconstruct the 3D angulation. The pipeline is illustrated in Figure 3 . First, we took two different views of X-ray images from the same patient, segmented the target forearm bones (radius and ulna), and extracted them separately (Section 2.2). Next, we fed the extracted bones to the style-transfer network and the rotational pose estimation network (Section 2.3); in parallel, we detected and measured 2D fractures (Section 2.4). Finally, we calculated the 3D fracture angle (Section 2.5).This retrospective, semi-synthetic study was HIPAA-compliant and approved by the institutional review board (IRB) of Duke University. We collected 1000 forearm X-ray images from the Duke University Health System (DUHS). The X-ray dataset contains 502 AP views and 498 LAT views (dataset X-ray). Annotation details are shown in Appendix D.For training the network for pose estimation, we acquired 22 additional CT images and manually segmented the 3D bones (ulna and radius). First, we registered these 3D bones into our defined same standard pose (details are provided in Appendix B), then generated 11,000 digitally reconstructed radiographs (DRR) as simulated X-ray images (dataset DRRpose) by rotating the registered bones around the x-, y-, and z-axes, including -π, π] to imitate the real-world situations. Obtaining fracture cases with X-ray and CT image pairs is difficult because such pairs are rarely acquired in clinical practice. To overcome this challenge, we utilized the Blender software (Community, 2018) to simulate and synthesize 40 fracture cases from 20 healthy CT exams, which were used to generate another 4000 DRRs (DRR-fracture-pose). Our synthetic fractures contain different conditions with various spins and displacements (shown in Appendix C). More details of the training/evaluation configurations are in Appendix H (Table H ).r x ∈ [-π, π]; r y ∈ [-π/6, π/6]; r z ∈ [Considering the necessity to separate two forearm bones with comparable texture, transparency, and potential overlap on an X-ray, we applied instance segmentation, combining detection and segmentation, for separating the two forearm bones. We employed the Maskscoring R-CNN (Huang et al., 2019) , which introduced another mask branch compared with Mask R-CNN (He et al., 2017) . When getting the segmented masks M r and M u from the input image I, we extracted the target bones with I r = I * M r , I u = I * M u and centralized the bones into the middle of the image. The segmentation is trained on dataset X-ray, and results and the bone extraction details are shown in Appendix E.To train a network for bone pose estimation without manual labels, we utilized the DRRpose and DRR-fracture-pose datasets. The network, Posenet, adopts a standard Resnet-18 as its core, with the output being a three-dimensional rotation parameter R = [r x , r y , r z ]. A T anh activation followed by a π scale is applied post-last layer to normalize the rotation parameters' range. Mean Squared Error (MSE) serves as the training's loss function.Our Posenets, trained on DRRs, encountered a challenge due to the differences between DRRs and real X-ray images, leading to potential drops in performance when applied to actual X-rays. To address this, we implemented a style-transfer technique using Cycle-GAN during the inference phase to bridge the gap between DRRs and X-ray images. This process involved using segmented bone images from X-rays as one domain and DRRs (DRR-pose and DRR-fracture-pose) as another, enabling the network to adapt X-ray images to the style of DRRs. Further details and results are in Appendix F.We utilized a Faster-RCNN (Girshick, 2015) network to detect fractures (Guan et al., 2020) .When training the fracture detection network, X-ray images (dataset X-ray) and DRRs (sampled from dataset DRR-fracture-pose) are combined to feed into the network, and the center of the detected fracture region was referred to as the "fracture point" or "breakpoint" to be detected. The measurements of 2D fractures consist of the following three parts: (1) Separate the input image I r or I u based on the fracture points;(2) for each view, fit two lines l i1 and l i2 using centerline extraction and line fitting; and (3) measure the angle θ 2d between the two lines l i1 and l i2 . This section's specifics are detailed in Appendix G.As illustrated in Figure 4 , and after previous steps, we got the line segments l 11 and l 12 for the image I 1 , and l 21 and l 22 for the image I 2 , and the rotational pose parameters (R 1 and R 2 ) for the radius bone in view 1 and view 2, I 1 and I 2 . We calculated the 3D fracture angle θ 3d = ∠(L 1 , L 2 ) using 3D geometry. The real bone centerline segments are Because l 11 is the projection of L 1 on plane P 1 and l 21 is the projection of L 1 on plane P 2 , we can infer that L 1 is at the intersection of plane P 21 and plane P 11 , thus the directional vector for L 1 can be calculated: L 1 = n 11 × n 21 ; Similarly, we can get L 2 = n 12 × n 22 . The 3D fracture angle, also known as the angle between three-dimensional lines L 1 and L 2 , can be calculated as follows:θ 3d = cos -1 L 1 ×L 2 ∥L 1 ∥∥L 2 ∥ . 
bone rotational pose estimation: 
 The bone rotational pose estimation was tested on two datasets. The first dataset was the DRR test set, which consists of 300 simulated X-ray pictures (dataset DRR-pose-test). This can be used to evaluate the performance of Posenet under varied rotational parameters.The second dataset includes 10 real X-ray images (dataset Xray-pose-test), which also have paired CT images (taken within 3 days). We also included a baseline as a comparison for bone rotational pose estimation (Krönke et al., 2022) , which used the segmented bone contours as the input of the pose estimation network.For the dataset DRR-pose-test, since its ground-truth rotational parameters have been known, we evaluated the model's performance by calculating the error between the predicted and ground-truth rotation angles on the x-, y-, and z-axes by the metric of L1-norm among all the available images. We evaluated the performance on the dataset Xray-pose-test by visualizing the registered bones (seen in Figure 5 ). We also analyzed the consistency of the object shape extracted from the X-ray and the projected DRR under the predicted posture by the shape similarity metric Hu-Moments Contours-match-I1 (Hu, 1962) . To evaluate our algorithms' effectiveness on 3D fracture angle assessments, we compared our method with two baselines: (1) Fracture-2D. This algorithm took only one view of images and measured the 2D fracture angle. This algorithm was similar to (Macken et al., 2022) , and detailed in Appendix G;(2) Fracture-orthogonal, which ideally assumed that the two input views are orthogonal to each other. The 3D fracture estimation was assessed using two datasets. The first dataset was the simulated DRRs (3D-fracture-generated-test), and the ground truth was measured manually by an average of two readers in our institution.Another dataset contains real-life clinical fracture cases, including patients with forearm fractures, noted as (3D-fracture-real-test), seen as Table H . 
results and discussion: 
 The data presented in Table 1 indicate that our pose prediction network can achieve less rotational error than just utilizing the contour information as previous work (Krönke et al., 2022) . These results show that for the prediction of the pose, the rotation along the x-axis, which is perpendicular to the projection plane and does not affect the 2D-fracture angle measured during projection, is the easiest to predict. The rotation along the z-axis, as the factor that most affect the fracture angle shown on X-ray images, as depicted in Figure 2 , is difficult to predict accurately due to the symmetry of the forearm bones concerning their cylindrical shape (Weinberg et al., 2016) . The posture estimation for the ulna exhibits fewer errors. Figure 5 shows the predicted pose of the radius and ulna based on the bones extracted from the input X-ray and the DRR generated from these predicted poses. The average distance between Hu-Moments for the 10 X-ray images is 0.75(±0.14) for the radius and 0.92 (±0.49) for the ulnas.The average predicted error for our method on the 3D fracture angle prediction is 3.42 degrees with a 95% confidence interval of [3.11, 3.73] for all 600 pairs of images, which is less than the method Fracture-2D with an average error of 8.92 and the method Fracture-orthogonal with an average error of 5.80. Compared to Fracture-2D and Fractureorthogonal, our technique provides more accurate fracture angle assessments (p < 0.03). The results of the angle measurements for all the image pairs are shown in Appendix I. (1) relying on a single X-ray to predict fracture angles is inadequate; (2) 2D measurements of fractures vary greatly with the bone's positioning, leading to unreliable outcomes; (3) using dual-view X-rays markedly enhances accuracy; and (4) our method significantly increase measurement precision. This improvement is most notable when the projection angles for the same bone (either radius or ulna) across two images span between 30 and 90 degrees, indicating nonorthogonal viewing angles, as detailed in Appendix I. Our algorithm's measurement of a 3D fracture in a real patient resulted in a 3D fracture angle of 13.01 • , which was just 2.06 degrees off from the manual measurements of 10.95 • , (seen in Appendix I, Figure 13 ). 
conclusion: 
 In summary, our research improves patient care for those with forearm fractures by addressing the challenge of inconsistent arm positioning in two X-rays. This inconsistency can lead to measurement errors and uncertainty for clinicians in distinguishing real changes in fracture angulation from artifacts due to varied positioning. Such uncertainties can complicate determining appropriate treatment strategies, potentially resulting in suboptimal care. Our algorithm uniquely accounts for variations in X-ray positioning, marking a novel approach in 3D reconstruction by considering objects' rotational posture. It enhances diagnostic accuracy, enabling clinicians to more reliably identify true changes in fracture angulation and reduce the risk of recommending unnecessary surgical interventions. We also acknowledged the limitations and discussed future directions, which are discussed in Appendix K.Figure 6 : Some examples of the well-posed and ill-posed cases; for the well-posed cases, the ulna and radius bones are nearly orthogonal between AP view and LAT view; and for the ill-posed cases, the rotations of the radius (ulna) between views are hard to predict through human's observation. the z-axis. Finally, we rotate the bone around the z-axis to make vector < F 1 , F 2 > in the direction of the x-axis, seen in the Figure 7 (bottom pipeline).The markers are set manually in 3D Slicer (Fedorov et al., 2012) , and the registration is done automatically through Python. The DRRs for a standard pose correspond to the equation R =< rx, ry, rz >=< 0, 0, 0 >, and the rotation parameters predicted by the pose prediction network (PoseNet) are all relative to this standard pose. had a perturb υ as predicted noise on n 2 :n 21 * = L 21 × (n 2 + υ) = L 21 × n 2 + (L 21 × υ) n 12 * = L 12 × (n 2 + υ) = L 12 × n 2 + (L 12 × υ) n 22 * = L 22 × (n 2 + υ) = L 22 × n 2 + (L 22 × υ) L 1 * = n 11 * ×n 21 * = n 11 × (L 21 × n 2 + (n 1 × υ)) L 2 * = n 12 * ×n 22 * = (L 12 × n 2 + (L 12 × υ)) × (L 22 × n 2 + (n 1 × υ)) = L 12 × n 2 × L 22 × n 2 + L 12 × υ × L 22 × n 2 + L 12 × n 2 × n 1 × υ + σ(•).(2)As our derived 3D angle is calculated, this noise might be accumulated to our final results θ 3d * = cos -1 L 1 * ×L 2 * ∥L 1 * ∥∥L 2 * ∥ . Based on the empirical results shown in Figure 12 , this error is more influential when θ = n 1 × n 2 → 1, where AP view and LAT are almost identical to each other. It is quite a rare case in real life. many 3D fracture angles are evaluated based on synthetic fractures with minimal real data. This is because we cannot seek eligible X-ray and CT pairs, and patients do not typically get both modalities simultaneously when fractures occur. We acknowledge this might bring limitations that the types of fractures covered in our study might not be comprehensive, such as the Comminuted Fracture (Rafi and Tiwari, 2023). We found this limitation acceptable because the comminuted fracture would require additional surgical treatment anyway, and the fracture angle measurements would less influence the diagnosis decision. Future expansion of the test set may involve creating phantoms that capture our image pairs. (3) Thirdly, our data was collected mainly on the Duke Health System, which would introduce some bias in the patient population distribution; we would release our codes in the future to get more potential applications and evaluations from different institutions. (4) Lastly, while our 3D fracture measurements are now limited to 3D angle measurements, one of the most important topics for orthopedic practice, we will expand our work to include bone misplacement and even 3D shape reconstruction.We put our preliminary analysis of measuring bone displacement, as shown in Figure 15 . Our current 3D angle estimation shown in Figure 4 mainly considers the angulation relationship between the bone and projection planes, using the plane norm vector. To add the displacement measurement in 3D, as shown in illustration Figure 15 here, we could add the measured distance of two break points in each projection plane (D 2D ), as well as combine the beam to receptor distance D s and object to source distance (D o ) to get the ratio of real distance to distance unit in the projected plane. We have two potential solutions for 3D bone shape reconstruction: one relies on an additional network for bone shape reconstruction, and another solely relies on similar mathematical solutions considering bone thickness when doing the 3D reconstruction (we currently only do centerline). 
 
 Appendix:  


Decision: Two of the three reviewers support accepting this work as a poster and believe it will generate meaningful discussions. I believe the rebuttal has addressed most of the important comments. The authors are requested to revise the camera-ready version in accordance with the feedback provided in the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper aims to obtain 3D fracture angle in the forearm from 2 2D bi-planar X-rays under the more realistic assumption that the arm might move during acquisition and the two views are not necessarily perfectly orthogonal. This paper proposes a pipeline for obtaining the 3D fracture angle via (1) segmentation of ulna/radius bone, (2) predict pose parameter of bones in 3D, (3) detect fracture location and measure fracture angle in 2D, and (4) compute 3D fracture angle from 2D fracture angle using pose parameters. The authors demonstrated that this pipeline results in more accurate 3D fracture angle prediction on both synthetic and clinical datasets.
main_review: main_review: I think this paper is good as is and would be a good contribution to this conference. 

I also want to comment on Reviewer xyYy (that gives a rating of 1). All weaknesses stated by this reviewer are related to writing & presentation rather than fundamental flaws of the work. I also disagree with the fact that this paper is badly written. In fact, this is one of the most clear paper I've read (best of 6 at MIDL).

Therefore, I'll keep my rating the same, i.e., 4.
strength_weakness: strength_weakness: + As the authors mentioned in limitations section, this pipeline is complicated with many moving parts, hyperparameters, etc.. It's arguable whether the improved performance justifies the added complexity.
questions: questions: Nothing
paper_summary: paper_summary: This paper discusses the limitations of traditional 2D X-rays in providing 3D information about imaged objects, particularly in measuring 3D fracture angles in the forearm. The paper proposes a method for reconstructing 3D skeletal structures using bi-planar X-rays, which involves addressing the unrealistic assumption that the target object does not move between the two images and that the two views are perfectly orthogonal. The paper also includes details on the training and evaluation of the network for pose estimation and the segmentation and extraction of forearm bones. The authors plan to expand their work to include misplacement of bones and 3D shape reconstruction.
main_review: main_review: This paper discusses the limitations of traditional 2D X-rays in providing 3D information about imaged objects, particularly in measuring 3D fracture angles in the forearm. The paper proposes a method for reconstructing 3D skeletal structures using bi-planar X-rays, which involves addressing the unrealistic assumption that the target object does not move between the two images and that the two views are perfectly orthogonal. The paper also includes details on the training and evaluation of the network for pose estimation and the segmentation and extraction of forearm bones. The authors plan to expand their work to include misplacement of bones and 3D shape reconstruction.

By supplementing necessary additions and complementing the draft based on the reviewer's comments, this article will be a stronger candidate for an accepted paper. I will sustain my initial evaluation, which was '5: Strong accept.'
strength_weakness: strength_weakness: The pape only focuses on the measurement of 3D fracture angles and does not include other important aspects of orthopedic practice, such as the misplacement of bones and 3D shape reconstruction. While the authors acknowledge this limitation and mention their plans for future expansion, it would have been beneficial to include some preliminary results or insights into these areas.

Another weakness of the paper is the reliance on synthetic fractures for evaluation. While the authors provide a clear justification for their use of synthetic data, it is important to note that synthetic data may not fully capture the complexity and variability of real-world fractures. Additionally, the authors mention that they cannot seek eligible X-ray and CT pairs, which raises questions about the generalizability of their method to real-world scenarios.
questions: questions: I strongly this paper to be accepted with oral presentation.
paper_summary: paper_summary: The authors propose a way to compute the angle of a fracture using two non-orthogonal planar radiographs. They collected a dataset comprised of 502 images of AP views and 498 images of LAT views.
main_review: main_review: I appreciate the authors' responsiveness in addressing all of my inquiries. They have indeed done commendable work in rectifying minor issues (although a typo still persists in Fig. 2 with "proection" instead of "projection"). While I still perceive the paper to lean more towards a technical report than a conference paper, the substantial improvement in the writing has influenced my final evaluation, prompting a change in my rating.
strength_weakness: strength_weakness: The paper reads more like a technical report. The authors mistakenly interchange the terms "X-rays" and "radiographs," with the former referring to the type of electromagnetic radiation that produces the latter.

There are several overly long sentences throughout the article, particularly noticeable in the third sentence of the abstract and the third paragraph of the Introduction.

Many of the figures are too small, making them difficult to interpret. In particular, Figure 3, which is crucial for explaining the proposal, is confusing. The section of the manuscript that references this figure does not adequately justify the choice of methods depicted.

Section 2.1, titled "Dataset Properties," does not appear to accurately present the properties of the dataset; instead, it describes how the authors prepared the datasets for training.

The experiments conducted are somewhat shallow and require improvement. Additionally, Table 1 is presented in an unconventional format.

Minor mistakes:

- A typo in Figure 1: 2D protection -> 2D projection
- Segmentor -> segmenter
- Untouched CT images -> ????
questions: questions: I would like to have a more detailed explanation of the motivations for the proposed framework.
Additionally, how about obtaining approval from the ethics committee?

Comments: title: Sustain my initial evaluation
Request help from area chair
Paper is open for discussions
Response to Reviewer kcx8.
Response to Reviewer xyYy.
Replies to the Reviewer's comments.
comment: By supplementing necessary additions and complementing the draft based on the reviewer's comments, this article will be a stronger candidate for an accepted paper. I will sustain my initial evaluation, which was '5: Strong accept.'
Dear Area Chairs:
 
I hope this message finds you well. I am reaching out to kindly request your assistance regarding the feedback we received from one of our reviewers, identified as reviewer xyYy. This reviewer’s score was “strong reject,” while the other reviewer’s scores were “strong accept” and “weak accept." While we believe that this reviewer is certainly entitled to their opinion that differs from others, the content of their review is not consistent with a “strong reject” score.
 
Their feedback, primarily focusing on minor aspects such as lengthy sentences, small figures, and terminology differences, does not seem to align with the severity of the score. We are concerned that this might have been an inadvertent error.

Furthermore, the reviewer's comments on our methodological approach were brief, with only half a sentence. This lack of detailed feedback has left us uncertain about how to address these concerns adequately.

Despite these challenges, we have made our efforts to address all the points raised by the reviewer, as well as provide a detailed explanation of the motivation for our work to help understand. However, we have not received any further communication or feedback, which adds to our apprehension about whether our response will be considered.

In this case, we would greatly appreciate your guidance on how to proceed to receive fair consideration of our work.
Thank you very much for your attention to this matter and your support in navigating this process. We look forward to your advice.
 
Best regards,
Authors from submission 277.
Dear Reviewers The authors have submitted their rebuttal addressing the raised questions. The paper remains open for further discussion and engagement.
Dear Reviewer, thanks so much for your comments, and we appreciate you liking our motivation to address real-world clinical challenges. Following the reviewer’s suggestions, we amended these points.

1. comment: ''The pape only focuses on the measurement of 3D fracture angles and does not include other important aspects of orthopedic practice, such as the misplacement of bones and 3D shape reconstruction. While the authors acknowledge this limitation and mention their plans for future expansion, it would have been beneficial to include some preliminary results or insights into these areas.'' ''One suggestion for improvement is to include more details on the limitations and assumptions of the proposed method. ''
 - **Answer**:  Following the reviewer’s suggestions, we amended the limitations and assumptions discussion (See updated Appendix K, limitations of our work) on these points.
     - We included an illustration how to reconstruct 3D displacements by expanding our algorithm.
     - We added insights and potential solutions to reconstruct shape as well.
     - We included discussions on other limitations of our study.

2. comment: ''Another weakness of the paper is the reliance on synthetic fractures for evaluation. While the authors provide a clear justification for their use of synthetic data, it is important to note that synthetic data may not fully capture the complexity and variability of real-world fractures.''
  - **Answer**: Thanks for pointing out the concerns. We add an additional discussions of the potential limitations about coverage of all real-life fracture types in our limitation section.

2. comment: ''For example, the authors could discuss the robustness of the method to different sources of noise and error, as well as the generalizability of the method to different view combinations and patient populations. "
  - **Answer**: We appreciate the reviewer’s suggestion here. 
     - We added an additional error estimation section to discuss the noise sources as well as our algorithm's robustness against these errors, as seen in Appendix J.
     - We added a demographic recap to our covered datasets and added it to our dataset section Appendix H, as shown in the newly added Table 3, to show the coverage of the population in our study. 
    - We also added additional discussions in the Limitation section to discuss the potential population bias. 
3. comment: ''Another suggestion is to provide more context on the clinical relevance of the proposed method.''
  -  **Answer**  Thank you for your recommendation! We have revised our manuscript to incorporate additional clinical motivations for our project into the introduction's first paragraph. Moreover, we have enhanced the conclusion to elaborate on the potential clinical advantages of adopting this method. Please review the updated manuscript for these modifications.

  Thanks again for your comments, and please let us know if you have further questions!
Thanks so much for the reviewer’s advice!

1. Comments: ''The authors mistakenly interchange the terms "X-rays" and "radiographs," with the former referring to the type of electromagnetic radiation that produces the latter.''
  - **Answer**: Thanks for pointing out the unclear usage of X-ray and radiographs. We revised all our teams into ‘X-ray images’ which refers to radiography gotten from rays.

2. Comments: ''There are several overly long sentences throughout the article, particularly noticeable in the third sentence of the abstract and the third paragraph of the introduction.''
  -  **Answer**: We revised our manuscripts to enhance readability, particularly by shortening the lengthy sentences found.

3. Comments: '' Many of the figures are too small, making them difficult to interpret. In particular, Figure 3, which is crucial for explaining the proposal, is confusing. The section of the manuscript that references this figure does not adequately justify the choice of methods depicted.'' 
-  **Answer**: Thank you for your feedback. We have made updates to the figures in our paper by: (1) enlarging the images and the font sizes displayed within the figures; (2) adding more detailed explanations for the figures both inside figures and captions; and (3) specifically for Figure 3, we have comprehensively revised it by using different colors to delineate various steps and to highlight which section corresponds to this part of the diagram.

4. Comments: '' Section 2.1, titled "Dataset Properties," does not appear to accurately present the properties of the dataset; instead, it describes how the authors prepared the datasets for training. ''
- **Answer**: Sorry for the confusion; we amended this section title to 'Dataset Preparation'.

5. ''Minor mistakes:
•	A typo in Figure 1: 2D protection -> 2D projection
•	Segmentor -> segmenter
•	Untouched CT images -> ????''
 - **Answer**: We amended all the listed typos here, as well as did another grammar check for the entire manuscript.

6. ''The experiments conducted are somewhat shallow and require improvement.''
  - **Answer**: We enhanced our experimental framework by incorporating an evaluation of the algorithm's robustness (see Section J),to expand the scope of our experiments. Please consult our revised paper for more details.

7. ''I would like to have a more detailed explanation of the motivations for the proposed framework.''
- **Answer**: We would like to explain the motivations in two aspects, (1) the motivation for solving the clinical task; (2) the motivation for proposing this algorithm and each step's decision making criteria. 
  - For motivation for this clinical task: 
    - When patients undergo follow-up X-ray imaging for a forearm fracture, replicating the exact arm position from previous X-ray images can be challenging. This difficulty can make it hard for clinicians to discern whether observed changes in fracture angulation are genuine or merely artifacts of repositioning. Sometimes, this ambiguity might lead to decisions for surgical intervention due to perceived 'loss of reduction' in the fracture, which could actually result from **different positioning** instead of **true clinical angulation change** [1]. Utilizing our algorithm can significantly minimize these positioning artifacts across X-ray images, potentially sparing patients from unnecessary treatments.
  - For motivation for this proposed framework.
    - As we acknowledge that we want to distinguish between **different positioning** and **true clinical angulation change**; we need to accomplish the following tasks in our algoritm (1) measure angles in 2D images; (2) predict the different positioning of bones; (3) combine these informations and reconstruct the real clinical fracture angle in 3D.
       - (1) measure angles in 2D images, which we chose to segment bones (Section 2.2), and detect where the fracture happens on bone and do 2D measuremnt in (Section 2.4);
       - (2) predict the different positioning of bones; For this task, we chose to build a Pose prediction network. To accomplish this, we generated DRRs to offer numerous cases without the need for human annotation; then, to make the network trained on DRRs only can also works for X-ray images, we included a Style-tranfer network (Section 2.3).
       - (3) Reconstruct 3D angle: from which we build our geometrical algorithm using the relationship of two Xray planes. 
       - We added more explanations in Method section about the motivations on each component of algorithm.

8.'' Additionally, how about obtaining approval from the ethics committee?''
 - **Answer**: Thanks for the advice, we updated the IRB approval information within our Dataset Preparation section.

9. ‘’Table 1 is presented in an unconventional format.‘’ 
 - **Answer**: Thanks for the advice. We updated table 1's format for better visualization.

Thanks again for your comments and Please let us know if you have further concerns!
Thanks so much for the reviewer’s insightful comments and advice! We addressed the following points:

1.	Comments: ''Grammar/flow of sentences could be improved, in particular in the experiments/results section''.

 - **Answer**: We have conducted a thorough review and proofreading of our entire manuscript, giving particular attention to the experiments/results section. Furthermore, we have updated the figures within our paper to improve clarity. Please review our revised manuscript.

2.	Comments: ''As the authors mentioned in the limitations section, this pipeline is complicated with many moving parts, hyperparameters, etc.. It's arguable whether the improved performance justifies the added complexity''.

 - **Answer**: Thanks so much for pointing out the necessity of this algorithmic complexity.
     - This problem comprises multiple components, and we find that all of which have to be solved to achieve clinically useful results.
         - Component 1: measure 2D fracture angles, which requires the extraction of bones and the detection of fracture points.
         - Component 2: Bone posture assessments, while forearm rotation is difficult to evaluate due to the underlying bones being round and a lack of reliable landmarks [1]. 
        -  Component 3: 3D fracture reconstruction, from which we build our geometrical algorithm.
    - For some aspects, we evaluated different choices with simpler solutions to see if they were good enough. 
        -  For example, we compared our intensity-included pose estimation (combine styler-transfer + pose estimation) with a previous contour-based algorithm (shown in Table 1). Our findings, detailed in Table 1, demonstrate superior performance by our method.
        -  For example, we evaluated our algorithm against a more straightforward reconstruction method based on orthogonal assumptions, which eliminates the need for bone rotational pose estimation or 3D geometrical inversion. As illustrated in the Appendix (Figure 12), this simpler method is over 97.2% more likely to produce larger errors than our algorithm and has a 32.1% likelihood of angulation assessment errors exceeding 10 degrees.
   - We found this improvement can have more clinical benefits, and the loss of accuracy would result in improper clinical decisions.
       -  When a fracture occurs, the choice of treatment depends on the degree of angulation and displacement of the fracture and the age of the patient; an angulation of less than 30 degrees and translation of less than 50% is generally accepted, whereas a higher degree of displacement is considered an indication for surgical intervention [2].
       - During a follow-up examination, clinicians might conclude that surgery is necessary for a patient due to a 'loss of reduction' in the fracture, indicating that the angles observed in the follow-up have not decreased satisfactorily [3].
       - This means, 10 degrees of error might bring about unnecessary surgical plans both in the fracture occurring stage and the recovery stage.
Given the complexity of the various components within this clinical problem, along with our supplemental analysis of the clinical advantages provided by this algorithm (which we have highlighted by updating the introduction and discussion sections of our manuscript), we hope the enhanced performance achieved through the algorithm could be justified.

Thanks again for your comments and Please let us know if you have further concerns or advice!

[1] Weinberg DS, Park PJ, Boden KA, Malone KJ, Cooperman DR, Liu RW. Anatomic Investigation of Commonly Used Landmarks for Evaluating Rotation During Forearm Fracture Reduction. J Bone Joint Surg Am. 2016 Jul 6;98(13):1103-12. doi: 10.2106/JBJS.15.00845. PMID: 27385684.	
[2] Macken AA, Eygendaal D, van Bergen CJ. Diagnosis, treatment and complications of radial head and neck fractures in the pediatric patient. World J Orthop. 2022 Mar 18;13(3):238-249. doi: 10.5312/wjo.v13.i3.238. PMID: 35317255; PMCID: PMC8935328.
[3]	Mehta N, MacFarlane RJ, Brown D. Traumatic disorders of forearm rotation: anatomy, biomechanics and treatment. Br J Hosp Med (Lond). 2014 Feb;75(2):72-7. doi: 10.12968/hmed.2014.75.2.72. PMID: 24521801.