DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering

Authors: zhongpai  gao
benjamin  planche
meng  zheng
xiao  chen
terrence  chen
ziyan  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:50:12

Abstract: Digitally reconstructed radiographs (DRRs) are simulated 2D X-ray images generated from 3D CT volumes, widely used in preoperative settings but limited in intraoperative applications due to computational bottlenecks. Physics-based Monte Carlo simulations provide accurate representations but are extremely computationally intensity. Analytical DRR renderers are much more efficient, but at the price of ignoring anisotropic X-ray image formation phenomena such as Compton scattering. We propose a novel approach that balances realistic physics-inspired X-ray simulation with efficient, differentiable DRR generation using 3D Gaussian splatting (3DGS). Our direction-disentangled 3DGS (DDGS) method decomposes the radiosity contribution into isotropic and direction-dependent components, able to approximate complex anisotropic interactions without complex runtime simulations. Additionally, we adapt the 3DGS initialization to account for tomography data properties, enhancing accuracy and efficiency. Our method outperforms state-of-the-art techniques in image accuracy and inference speed, demonstrating its potential for intraoperative applications and inverse problems like pose registration.

Main paper: 
 introduction: 
 Motivation. Digitally reconstructed radiographs (DRRs) are simulated (in silico) 2D X-ray images rendered from 3D computational tomography (CT) volumes. While DRRs are widely utilized in preoperative settings, such as optimizing dose delivery and in radiation oncology [20, 41, 6] , their potential intraoperative applications remain underexplored due to computational bottlenecks and naive modeling capability [1, 27] . For instance, real-time multimodal registration for image-guided procedures is currently impractical because of the time-consuming process of generating DRRs and integrating them into slice-to-volume registration [36] .The advent of GPU-accelerated computing has significantly improved the efficiency of radiography simulators [3, 39, 5] , though they still fall short of meeting the stringent requirements of real-time applications. Recent developments in inverse graphics have led to the creation of differentiable DRR renderers (e.g., DiffDRR [11] , X-Gaussian [7] , GaSpCT [25] ), which show promise for inverse problems such as 2D/3D CT image registration [12] . However, these methods do not fully capture the complexities of X-ray image formation. They typically use computationally efficient ray-tracing techniques that model the attenuated photon fluence at each detector pixel by accumulating Hounsfield unit (HU) values along the 3D view-ray through the voxelized CT volume. This approach, while fast, cannot account for complex noise-inducing physical effects such as Compton scattering and beam hardening [14] , crucial for accurate X-ray imaging. Physics-based Monte-Carlo simulations [2, 15, 3, 4] do not suffer from these limitations. Relying on the decomposition of CT volumes into realistic materials via HU thresholding, they accurately simulate single-photon transport and probabilistically evaluate photon-matter interactions (photoelectric effect, Compton scattering, Rayleigh scattering, etc.) to determine the final attenuation and hit-region of each photon. This approach provides a more accurate representation of X-ray image formation; but the computational intensity of these methods-requiring >10 8 iterations to render an image-makes them impractical for real-time applications and inverse problems.In this paper, we propose a novel, more elegant balance between (a) realistic physics-inspired X-ray simulation and (b) efficient, differentiable DRR generation, as illustrated in Figure 1 .Contributions. Our approach leverages 3D Gaussian splatting (3DGS) [17] for efficient DRR generation from CT volumes, similar to X-Gaussian [7] and GaSpCT [25] . However, we introduce a crucial adaptation: our direction-disentangled 3DGS (DDGS) method accounts for the dependencies of X-ray image formation on light directionality. Specifically, we decompose the radiosity contribution of 3D regions into two independent components: a non-directional component modeling the isotropic X-ray interactions with the corresponding material, and a direction-dependent component approximating higher-dimensional anisotropic interactions, such as scattering. Unlike the more generic view-dependent spherical-harmonics decomposition adopted by common 3DGS solutions [10] , ours is tuned to the specific behaviors of high-energy photons. This formulation allows our method to implicitly learn and reproduce the effects of scattering on X-ray images without the need for complex photon-transport simulations at runtime. Additionally, we adapt the initialization phase of 3DGS to account for the properties of tomography data. We adopt a geometry-and intensity-based dual strategy to sample 3D points that correspond to: (a) contact regions between different anatomical elements, where photon transport is likely to show discontinuities; and (b) homogeneous regions, where the primary attenuation contribution can be sparsely modeled but could benefit from view-dependent noise modeling.We demonstrate that our solution achieves higher image accuracy compared to state-of-the-art methods across various medical benchmarks Additionally, we showcase the efficient application of our method to inverse problems, such as pose registration, highlighting its potential for real-time intraoperative use cases. Our DDGS achieves better registration accuracy and runtime performance than analytical DRR methods.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction explicitly list our contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 The generation of DRRs and the simulation of X-ray images have been subjects of extensive research due to their critical roles in medical imaging, particularly in radiotherapy planning and diagnostic imaging [20, 41, 6, 43] . In this section, we provide some insight on X-ray imaging, traditional simulation tools, and the recent integration of machine-learning techniques.X-ray Image Simulation. X-ray imaging relies on the interactions of high-energy photons-emitted by the X-ray source of the scanner-with the matter (e.g., body part) placed in between the X-ray source of the scanner and its radiation-sensitive detector cells (dexels). Imaging contrast arises from differential absorption and scattering of X-ray photons by tissues, primarily through the photoelectric effect, Compton scattering, and, to a lesser extent, beam hardening for polychromatic scanners. The photoelectric effect involves the absorption of an X-ray photon by an atom and is more pronounced in higher atomic number materials like bone, causing them to appear whiter on X-ray images2 ; whereas Compton scattering occurs when an X-ray photon collides with an outer electron, resulting in the photon being deflected and losing energy, which contributes to image noise and reduced contrast [14, 29] .A variety of Monte Carlo (MC) simulation suites [2, 15, 3, 4] model these X-ray interactions and light transport. They trace the paths of a large number of individual photons as they undergo probabilistic interactions within the material. By simulating numerous photon trajectories, Monte Carlo methods account for the random nature of photoelectric absorption and Compton scattering, as well as secondary effects such as fluorescence and electron transport. These MC solutions produce highly realistic simulations of radiographic images, though at a high computational cost which makes them inadequate for most DRR use-cases. Furthermore, for these tools to render realistic DRRs from CT volumes, one must first pre-process the CT data to assign proper materials and their corresponding physical properties to each voxel, which is essential for realistic simulation. This step is often approximated by decomposing the volume into a small set of predefined materials according to user-provided HU ranges (e.g., "air" material assigned to voxels with HU value < -800, "lung" assigned to voxels in range ] -800, -200], "fat" for ] -200, -100], etc.). This process can be time-consuming and prone to errors, affecting the fidelity of MC-based DRR generation [9] .Efficient and Differentiable DRR Generation. Analytical DRR rendering methods [39, 38, 5] have been developed in parallel to the aforementioned MC suites for real-time or near-real-time applications. These methods have a much lower computational footprint, as they do not simulate individual photon interactions but instead use mathematical models to directly calculate the primary paths and approximated attenuations of X-rays. The seminal work of Siddon [31] on computing line integrals through a discretized CT volume is still at the core of most recent DRR solutions, e.g., which proposed differentiable formulation of the rendering steps [11] or integrated neural networks to add realistic noise to the output [35] . However, the inherent approximations in these methods can affect their accuracy, particularly in modeling artifacts present in heterogeneous tissues or complex anatomical structures, e.g., caused by scattering (non-primary light contributions).Recently, researchers [7, 25] have tried to apply 3D Gaussian splatting (3DGS) techniques [17] to DRR rendering, i.e., optimizing a cloud of 3D Gaussians to approximate voxel data, thereby enabling faster rendering with minimal accuracy loss. However, these early solutions overlooked the specific nuances of X-ray imaging, directly applying 3DGS methods designed for natural imaging [25] or oversimplifying the physical properties of X-ray attenuation [25] , leading to suboptimal modeling.In this paper, we propose a novel formulation of 3D Gaussian splatting, tuned to more efficiently approximate X-ray imaging (time-and quality-wise).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please refer to Section 5 in the main paper for detailed discussion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Gaussian Splatting. 3DGS [17] is a point-based rendering technique using 3D Gaussians to explicitly represent scenes in a more compact manner than volumetric representations. Each Gaussiang i = {µ i , Σ i , α i , f i } is defined by its mean position µ i ∈ R 3 , covariance matrix Σ i ∈ R 3×3 (usually decomposed into separate scale and rotation parameters), opacity α i ∈ R, and radiance properties f i ∈ R k , where k depends on the light contribution model adopted. E.g., k = (L + 1) 2 × 3 for anisotropic, view-dependent RGB radiance decomposed into spherical harmonics (SH) of degree L, a model commonly used for natural-imaging applications. Images are rendered by casting view-rays through each pixel p into the scene and alpha-blending the Gaussian contributions to the final ray color C, as:EQUATIONwhere n is the number of Gaussians considered, c j is the radiance (computed from f j , e.g., via SH) of the jth Gaussian on the ray, and µ j and Σ j are the image-plane projections of µ j and Σ j .Since the aforementioned rasterization process is differentiable w.r.t. to the Gaussian parameters, the 3DGS representation of a scene can be learned via gradient-descent, given a set of 2D scene observations I GT and their corresponding camera parameters. A combination of ℓ 1 and SSIM [40] loss functions are commonly-adopted as criterion for this iterative optimization process [17] :EQUATIONwith I the rendered images (based on provided camera parameters), and λ a loss-weighting hyperparameter. The initial state is typically obtained by sampling relevant 3D points in the scene domain for the Gaussians, e.g., from a structured-from-motion (SfM) point-cloud [34, 28] .Application to DRR. The volumetric rendering performed by 3DGS methods is conceptually similar to the one performed by analytical DRR methods [31, 11] , i.e., aggregating the attenuation values of light-rays cast through the voxelized CT volume. This similarity has recently motivated researchers [7, 25] to apply 3DGS models to DRR applications, driven by the compactness of 3DGS representations compared to voxel data (addressing the O(n 3 ) complexity inherent to voxel rendering). By optimizing a 3DGS model to approximate the visual properties of the CT volume, the resulting representation can render accurate DRRs much faster than traditional methods, making it more suitable for real-time intra-operative visualization. Consequently, the use of traditional, slower rendering solutions can be limited to generating DRR targets (I GT ) for the offline (i.e., pre-operative) training.However, these early 3DGS-for-DRR solutions [7, 25] do not account for noise-inducing photon interactions (e.g., scattering) when applying the analytical methods [30, 5] to render their training data. Since scatter-free DRRs are not affected by ray directions (i.e., the attenuation at each point is naively considered isotropic), Cai et al. [7] simplified the radio-intensity function of each Gaussian to c i = sigmoid (b • f i ), where b ∈ R k is a direction-independent optimizable basis vector shared by all splats. Furthermore, their Gaussian point-cloud is initialized via evenly-spaced Cai et al. [7] or random-uniform [25] sampling of the scanned 3D space, i.e., ignoring relevant geometrical and material properties of the target CT data. Inadequate initialization strategies negatively impact the convergence of 3DGS models and the resulting image accuracy [16] . In this work, we propose to reformulate both the initialization strategy and the radiance function of the 3D Gaussians to address these limitations.We argue that, while X-ray scattering models are incompatible with the 3DGS rendering formulation (secondary rays would have to be cast, increasing the computational footprint exponentially), their anisotropic impact on X-ray imaging can be approximated to some extent by direction-dependent radiance functions. However, modeling this high-dimensional residual contribution is only meaningful if it appears in the training DRRs, i.e., if the target radiographs are rendered using physics-inspired simulation tools. This is not always possible, e.g., if the mapping from HU values to materials for the target CT scans is not provided. Ideally, in such cases, when only scatter-free DRRs can be generated, the radiance function should gracefully degrade to a lighter anisotropic formulation (e.g., similar to [7] ), to avoid unnecessary computations that could impact the model optimization, as well as its runtime performance. Moreover, X-ray interactions with matter themselves involve both isotropic (e.g., photoelectric absorption/fluorescence, Rayleigh scattering) and anisotropic (e.g., Compton scattering) phenomena [14, 22] . Therefore, for the sake of both modularity and accuracy, we propose to decompose our DRR representation into isotropic Gaussians g iso i and anisotropic, direction-dependent ones g dir j . We formulate their respective radiosity functions as:EQUATIONwhere f iso i , f dir j ∈ R k are feature vectors respectively encoding the contribution of isotropic and anisotropic Gaussians, b iso ∈ R k is a global isotropic basis vector (similar to [7] ), and B dir ∈ R k L ×k is a second, anisotropic basis matrix that interacts with the L-degree spherical-decomposition Y : R 2 → R k L applied to the ray angles θ, ϕ without constant term (degree = 0), resulting in It is important to note that, while the usual spherical-harmonics representation for Gaussian splatting also contains isotropic (degree = 0) and anisotropic (degree > 0) components, the model assumes that both contributions are co-located (i.e., they belong to the same 3D Gaussian g i of position µ i and covariance Σ i ). To account for the complexity of X-ray light transport, we relax this co-location constraint, i.e., modeling isotropic and anisotropic contributions via distinct 3D Gaussians (g iso i and g dir j ). We demonstrate in our evaluation that this disentanglement results in both higher image quality and lighter representation.k L = L(L + 2).Our second main contribution targets the initialization of the point-cloud that supports the Gaussianmixture modeling of the 3D data. Since 3DGS was defined for natural imaging (e.g., ignoring sub-surface light transport), common initialization strategies limit their sampling to scene-surface points (SfM- [17] or depth-guided [24] subsampling). Such sampling of the 3D space is, however, inadequate for volumetric data, ignoring possibly salient regions. While uniform sampling of the CT space could be considered [7, 25] , we argue that such a strategy is also suboptimal for anatomical data, discarding domain-relevant properties. We demonstrate empirically (c.f . Section 4) that uniform sampling results in slower convergence of the 3DGS representation and degraded runtime performance.Therefore, we propose a novel twofold sampling strategy that accounts for the specific nature of CT data and for our dual isotropic/directional 3DGS model. Based on the radiodensity values contained in the target CT volumes, our solution samples two set of 3D points P mc ∈ R n1×3 and P dw ∈ R n2×3 with distinct, complementary distributions (n 1 , n 2 scalar hyper-parameters).Not unlike [24] , the first set of points is sampled by applying the marching-cubes algorithm [21] to extract points at the interface between materials with different physical properties (i.e., distinct HU responses in the CT volume). These interface regions typically need careful modeling when simulating X-ray imaging, as they result in discontinuous photon transport.To complement this first set of points, the second set is semi-randomly sampled from the voxel centroids, according to a uniform distribution weighted by the voxels' radiodensities. I.e., voxels with higher radiodensity, contributing to the X-ray attenuation, are more likely to be picked to initialize the Gaussians. In other words, this radiodensity-weighted strategy identifies additional points within the homogeneous regions of the CT volume that also contributes to the imaging process.Based on their distinct properties, we assign the entire P mc to initialize the isotropic Gaussians, whereas we equally split P dw to initialize both isotropic and direction-dependent Gaussians. All in all, our dual radiodensity-aware strategy improves the initialization of the 3DGS representation and is more explainable than prior work (see Figure 2 ).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Preliminaries are provided at the beginning of Section 3, and further theoretical background is provided in the supplementary material.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental protocol: 
 Implementation. For evaluation, we set the decomposition degree to L=1, the feature dimension to k=8, and initial cloud sizes to n 1 =15,000 and n 2 =10,000. We apply a loss weight λ = 0.2. We choose the default threshold level (i.e., the average of the minimum and maximum volume) for the marching-cubes algorithm. DDGS training is performed on a single NVIDIA A100 using an Adam optimizer [18] with a learning rate of 1.25 × 10 -4 for b iso and B dir and 2.5 × 10 -3 for f iso and f dir . Default 3DGS [17] learning rates are applied to the remaining parameters.Datasets. We consider 4 datasets. (1) NAF-CT [42] includes four CT images of abdomen, chest, foot, and jaw. For each, we adopt TIGRE [5] to sample 50 evenly-distributed projections for training and 50 randomly-distributed ones for testing, in the range of [-90 • , 90 • ] (scatter-free DRRs).(2) CTPelvic1K [19] is a large dataset of pelvic CT images. We consider the first 10 scans of the sub-dataset6 in our experiments. We adopt DeepDRR [35] to generate 60 training (evenly-distributed) and 60 (randomly-distributed) testing DRRs (in the range of [-60 • , 60 • ]), taking advantage of the scattering-modeling capability of DeepDRR to render realistic X-ray data. (3) Ljubljana [26] is a clinical dataset of 10 patients undergoing neurovascular surgery. Each patient underwent one CT and two X-ray angiography scans. The pose of each X-ray is also provided. Following DiffPose strategy [12] , we randomly sample 900 projections for training and 100 for testing. ( 4) Finally, we provide some additional qualitative results on DeepFluoro [13] , a collection of pelvic X-rays and CT images from six cadavers.Metrics. We evaluate our method in two settings: novel-view synthesis and intraoperative 2D/3D image registration (i.e., pose estimation). For novel-view synthesis, we use the standard PSNR and SSIM as metrics. For intraoperative 2D/3D image registration, we consider the error both in terms of rotation (angular distance) and translation (Euclidean distance). We further measure the clinicallyrelevant registration accuracy of key anatomical landmarks for Ljubljana scans [26] where they are provided. Following [12] , we compute the target registration error (TRE) w.r.t. the positioning of these landmarks after registration.We first validate our contributions in terms of the quality of rendered DRRs and compactness of our dual Gaussian-mixture representation.Comparative Evaluation. We compare to 3DGS [17] and X-Gaussian [7] on the NAF-CT [42] and CTPelvic1K [19] datasets for the novel-view synthesis. For each method, we adopt the hyperparameters recommended by their respective authors (e.g., L = 1 and 3 for normal 3DGS [17] and k = 8 and 32 for X-Gaussian [7] ).As shown in Table 1 , for NAF-CT [42] (without scatter-related noise), our DDGS outperforms both 3DGS and X-Gaussian in terms of PSNR and SSIM. For CTPelvic1K [19] (with scatter), our DDGS significantly outperforms 3DGS and X-Gaussian in both PSNR and SSIM while using considerably Furthermore, we compare our DDGS (with degrees L = 1 and L = 3 and feature dimension k = 8) with traditional 3DGS (degrees L = 1 and L = 3) and X-Gaussian (feature dimensions k = 8 and k = 32) on the 001 and 002 data of CTPelvic1K [19] at iterations of 500, 2000, 7000, 15,000, and 30,000, as shown in Table 2 . Our model with L = 1 and k = 8 performs comparably to 3DGS with L = 3 and outperforms X-Gaussian with k = 32 after 15,000 iterations. Experiments on more data can be found in Table S1 . Our model with L = 3 achieves the best performance overall. Figure 3 presents a qualitative comparison of the testing set, illustrating both the synthetic views and the generated 3D Gaussian points.It should also be highlighted that, even though both the original CT scan (as voxel grid) and our 3DGS-based solution are explicit representations, the latter is significantly more compact. E.g., our model can represent a CTPelvic1K scan with only 42,625 Gaussians (c.f . Table 1 ), defined by 19 float values each (3D position/rotation + 3D covariance + opacity + feature vector) and a shared 32-dimensional basis vector b, so 809,907 float values in total; whereas the original CT scans are each composed of 512 × 512 × 500 = 93, 363, 200 values. Indeed, the voxel data may contain large homogeneous regions, which can be approximated with few Gaussians, hence a significant compression rate.Finally, similar to [12] , we qualitatively compare the results of recent DRR methods with actual X-ray images, for datasets providing both CT scans and real, posed projections (DeepFluoro [13] , Ljubljana [26] ). Results are shared in Figure 4 , with our method achieving state-of-the-art PSNR. Further analyses are provided in Appendix B.Ablation Study. Table 3 demonstrates the effectiveness of our direction-disentangled representation for 3D Gaussians. For several settings, we observe performance degradation when learning the direction-dependent and direction-independent components in the same 3D Gaussians (directentangled), learning the direction-independent 3D Gaussians alone (direct-independent), or learning the direction-dependent 3D Gaussians alone (direct-dependent). Additionally, we decouple the direction-dependent and direction-independent components in 3DGS [17] (3DGS-disentangled), which also showed lower performance compared to our learnable features. Figure 5 shows the rendered views from disentangled 3D Gaussians, highlighting that the direction-dependent 3D Gaussians focus more on bones or anatomical structures, while the direction-independent 3D Gaussians focus more on the background.Table 4 shows the effectiveness of our proposed initialization scheme (radiodensity-aware) compared to other initialization methods. Our scheme outperforms both the uniformly random sampling used in 3DGS [17] and the even sampling method used in X-Gaussian [7] .At last, we conduct experiments on the impact of feature dimension k, as shown in Table 5 . Our results indicate that increasing the feature dimension generally improves overall performance. However, the improvement becomes marginal once the feature dimension exceeds k = 16. We validate our solution in terms of applicability to downstream tasks, considering here the registration of intraoperative 2D X-ray images to preoperative 3D CT volumes; a task crucial to a variety of clinical applications.We choose the first five testing images on the 001, 002, and 003 CT data of CTPelvic1K [19] for image registration. Considering this task as an optimization-based problem that can be solved via inverse graphics (i.e., rendering synthetic images based on predicted poses and comparing to the target image, then backpropagating the difference to improve the pose prediction), we adopt the framework from iComMA [32] , replacing their image rendered by DDGS. We use Adam optimizer with a learning rate of 0.05. We use the isocenter pose as the initialization pose. As shown in Table 6 and Figure 6 , our DDGS achieves the lowest rotation and translation errors.Finally, we also consider the experimental protocol proposed in DiffPose [12] , where the authors rely on a pretrained pose-regression CNN to get a first rough estimation of the scanner pose, before refining said pose via a gradient-descent based optimization, leveraging their differentiable DRR renderer [11] . We adopt their pose-initialization network and testbed for the Ljubljana dataset, replacing their DiffDRR analytical renderer [11] by our DDGS, and compare to them in terms of runtime performance and landmark-registration accuracy. The results, reported in Table 7 , highlight the convergence boost brought by our lighter solution.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Implementation and experimental details are provided, so that an expert in the art should be able to reproduce the presented results. 
conclusion: 
 We proposed DDGS, a novel 3DGS-based representation for efficient and realistic DRR generation from CT volumes. We extended traditional 3DGS by considering light directionality, crucial for accurate X-ray image modeling. By decomposing radiosity into non-directional and directiondependent components, DDGS captures isotropic interactions and anisotropic effects. Unlike generic view-dependent spherical-harmonics decomposition, DDGS implicitly learns scattering effects, substantially improving the speed and accuracy of downstream clinical applications such as intraoperative 2D/3D registration.Limitations. It should be noted that, similar to previous methods, our model expects an offline DRR renderer to produce realistic ground truth to train DDGS, which can be a limitation for some scenarios (e.g., when such renderer or certain parameters required for realistic rendering are not available). Our direction-dependent function relating to anisotropic X-ray effects may also fail to approximate complex multi-bounce light scenarios. Recent developments in N -dimensional Gaussian splatting [8] show promising potential for modeling such cases. We should also note that the quality of DRRs is bound to the quality of the input CT scans. This issue affects both DRR methods and Monte-Carlo simulations (as the latter relies on the segmentation of CT scans into materials-specific regions -if a 3D scan is noisy, so will the material segmentation and simulation results). When our method uses simulators to get target images, it will face similar noise issues, though, our experiments show that DDGS barely introduces additional noise. Compensating for CT noise is an interesting, under-explored research direction that could benefit all DRR methods; but, we believe it is beyond the scope of our study.Societal Impact. The above limitations may restrict the adoption of DDGS to specific use cases, and our method should undergo a clinical evaluation of the generated images in terms of anatomical accuracy. Nonetheless, we believe that our proposed method can positively impact both the computer vision and medical communities by providing a more efficient and versatile DRR tool. By having 2 sets of Gaussians with distinct yet complementary contribution functions (isotropic versus direction-dependent functions), our solution can better model DRR imaging. During optimization, each Gaussian set will approximate its respective imaging effect, conditioned by its respective function.We randomly split P mc (3D points sampled based on the CT scan distribution) into 2 sets, each subset contributing to the initialization positions of g iso i and g dir j (half the 3D points in P mc will serve as initialization for g iso i , along with the entire P mc set; and the other half of P mc will serve as initialization for g dir j ). After initialization, during 3DGS-based model optimization, the points in each set of Gaussians can move/split/drop independently (along with the optimization of their respective covariance/opacity/feature values). Since the optimization of each set (isotropic and anisotropic) is conditioned on their respective differentiable absorption function, they will acquire distinct properties.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: All data used are public, and implementation release is pending approval at the moment.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] .Justification:See Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: All standard guidelines were followed. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please refer to Section 5 of the main paper for further details. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Public datasets and libraries are referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To further support the results in Table 2 and Table 3 , we evaluate our method on all the 10 selected data from the CTPelvic1K dataset, as shown in Table S1 .Table S1 : Novel-view synthesis evaluation on CTPelvic1K data. and Plastimatch [30] ; [7] to other NVS baselines). We found that only DiffPose (based on DiffDRR) [12] provides a qualitative comparison to some real scans.(L = 1, k = 8) 3DGS (L = 1) X-Gaussian (k = 8) DDGS (ours) (L = 3, k = 8) 3DGS (L = 3) X-Gaussian (k = 32) direct-entangled c.f . Ablation Study PNSR SSIM PNSR SSIM PNSR SSIM PNSR SSIM PNSR SSIM PNSR SSIM PNSR SSIMQuantitative comparison to real data is challenging. Not all imaging parameters are usually provided or accurate enough to create matching DRRs (w.r.t. intensity, CT pose, etc.). It is especially hard to render DRRs aligned with real scans with existing Monte-Carlo (MC) simulation tools [2] [3] [4] 15] ; which is likely why no DRR paper has performed such evaluation. Their interfaces and custom coordinate conventions are not compatible with pose annotations in public CT/X-ray datasets, and their documentation/support is lacking. Bridging the convention gap between MC and analytical communities would greatly benefit our domain and could be the focus of our next effort.Nevertheless, we do provide an indirect comparison to real projections. E.g., the registration experiments (Table 6 ) implicitly provide insight into the similarity between real and synthetic data, as pose optimization is done by comparing DRRs to real target scans.In this supplementary, we share a more direct comparison. We use our DDGS-based registration method to refine GT poses in Ljubljana data (c.f . aforementioned GT inaccuracies), then use the refined poses to render and compare DRRs to real scans. Results can be found in Table S2 and Figure 4 . E.g., when zooming into the images of the latter figure, one can observe that DiffDRR suffers from pixelization, unlike GS methods.NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.X-ray film is historically white, turning black when exposed to high-energy photon. 


Decision: The manuscript introduces the Direction-Disentangled Gaussian Splatting (DDGS-CT) method, aimed at improving the balance between realistic X-ray simulation and efficient DRR generation using 3D Gaussian Splatting (3DGS). The method distinguishes itself by decomposing the radiosity into isotropic and direction-dependent components, which approximates complex anisotropic interactions without runtime simulation. The proposed technique demonstrates significant improvements in PSNR and SSIM for DRR renderings of multiple organs, suggesting superior performance compared to existing methods.

Strengths : 
The DDGS-CT method introduces a novel technique for balancing efficiency and realism in DRR rendering, which is a significant step forward in medical imaging.
The method models isotropic and anisotropic contributions via distinct 3D Gaussians to handle complex X-ray interactions like Compton scattering. 
It demonstrates substantial improvements in PSNR and SSIM scores, validating its effectiveness over existing methods.

Weaknesses :
The authors have adequately discussed their limitations, but some concerns remain about the robustness and reliability of the method in clinical practice, particularly regarding artifacts and the amplification of errors.

Recommendations:
The authors could discuss the results a little more to show the effectiveness of their method in terms of benefits which sometimes seem marginal. For example, the notion of noise or imprecision in the initial segmentation concerns all methods in any case.

The authors have responded fully and honestly to the reviewers' many comments. They have provided additional information, tables and figures to improve the presentation of their results.

Reviews: review: paper_summary: paper_summary: This manuscript presents a novel method called Direction-Disentangled Gaussian Splatting (DDGS-CT), tailored for balancing realistic X-ray simulation and efficient DRR generation using 3D Gaussian Splatting (3DGS). It addresses the challenges posed by intricate physics computation, which often harms the application of Monte Carlo simulations. The method decomposes the radiosity into isotropic and direction-dependent components to approximate complex anisotropic interactions without runtime simulation. Notably, it achieves significant improvements in PSNR and SSIM on DRR renderings of multiple organs, demonstrating superior performance.
strength_weakness: strength_weakness: 1.  “… we equally split P^dw to initialize both isotropic and direction-dependent Gaussians.” I’m confused about how to split the points set P^dw into disentangled isotropic and anisotropic point sets. The authors should provide a more detailed explanation.
	2. How do the disentangled isotropic and anisotropic 3DGS be rendered? Based on Figure 1, it seems that the two disentangled 3DGSs are integrated to render the output. If so, why should they be disentangled? The authors should explain the motivation for this design.
	3. In the Experiments, the 3D Gaussian-based methods are evaluated, yet the DRR-based methods mentioned above are not validated both in qualitative and quantitative comparison. Can they be added?
questions: questions: 1.	I wonder if it is necessary to disentangle the isotropic and anisotropic point sets since they are both splatted to render the output images. I think if they are disentangled, it would be more reasonable that they are also rendered using different algorithms.
2.	As far as I know, 3DGS is actually an explicit representation of 3D data. I wonder if it is necessary to transform 3D data from an explicit representation (volume) into another explicit representation (3DGS).
limitations: limitations: The authors have adequately discussed their limitations.
paper_summary: paper_summary: This paper proposes DDGS, a Gaussian splatting (GS) based method for rendering realistic 2D X-ray images from 3D CT volumes. Taking advantage of GS, the proposed method operates in real-time. Moreover, the DDGS model employs isotropic Gaussians and anisotropic, direction-dependent Gaussians to model complex X-ray physical interactions, such as Compton scattering. The proposed method is well-motivated and technically sound. The effectiveness of the DDGS model is confirmed on multiple datasets.
strength_weakness: strength_weakness: **Effectiveness of the Radiodensity-Aware Dual Sampling**

From Table 4, the improvements made by the Radiodensity-Aware Dual Sampling are marginal (<0.3 dB), so its effectiveness needs further investigation.

**Lack of CT volumes**

Although the purpose of this work is digitally reconstructed radiographs, showing the CT volume represented by GS is necessary for evaluating the proposed model.

**Visualization results**

The qualitative results shown in Figure 3 are hard to distinguish visually. Please consider showing the error maps between the reconstructions and the ground truth.

**Some typos**

For example, in line 90, “] − 800, −200], “fat” for ] − 200, −100]” should be “[ − 800, −200], “fat” for [ − 200, −100]”.
questions: questions: See the section of Strengths and Weaknesses, please.
limitations: limitations: The authors mention enough limitations of their work.
paper_summary: paper_summary: Digitally reconstructed radiographs (DRRs) are simulated 2D X-ray images generated from 3D CT volumes, widely used in preoperative settings but limited in intraoperative applications due to computational bottlenecks. In this paper, the author proposes an approach that combines realistic physics-inspired X-ray simulation with efficient, differentiable DRR generation using 3D Gaussian splatting (3DGS). This method excels in both image accuracy and inference speed, demonstrating its potential for intraoperative applications and inverse problems like pose registration.
strength_weakness: strength_weakness: 1. The methods used for comparison are limited and do not reflect the current state of the field. The state-of-the-art methods claimed by the author are not widely accepted as state-of-the-art by researchers, and some have not been peer-reviewed, like reference [7,11]. Therefore, I believe the experimental results lack persuasiveness.
2. The structure of the paper is not well organized. Despite the author clearly stating the motivation and contributions, the intentions of the author remain unclear. I have a rough idea of what the related work section is trying to convey, but its organization is also poor. It is recommended to include the 'Application' from section 3.1 within the related work.
3. In the ablation experiments, only a portion of the data was tested. Results from testing the entire dataset would be more convincing.
questions: questions: 1. The methods section lacks explanations for some symbols. Please review and provide explanations where necessary. Additionally, are g_i^{iso} and c_i^{iso}, as well as g_i^{dir} and c_i^{dir}, referring to the same concepts? Please review carefully. For the methods section, it is recommended to include an overview diagram for clarification.
2. Add more true sota methods for comparison.
limitations: limitations: 1. The experiments are insufficient. The comparison method lacks representativeness and fails to demonstrate the true level of the field.
2. The method description is confusing and the contributions are not clear.
paper_summary: paper_summary: 1. The proposed direction-disentangled 3DGS (DDGS) method decomposes the radiosity contribution into isotropic and direction-dependent components, able to approximate complex anisotropic interactions without complex runtime simulations. specifically, it modeling isotropic and anisotropic contributions via distinct 3D Gaussians.

2. The paper clearly mentioned that 3DGS-for-DRR solutions [30, 11] do not account for noise-inducing photon interactions (e.g., scattering) when applying the analytical methods [29, 5] to render their training data.
strength_weakness: strength_weakness: 1. Although Gaussian Splatting is widely used in natural color scenes, this paper does not provide a detailed mathematical description of the "migrated version" of Gaussian Splatting in combination with the physical scene of CT.  Instead, it directly use the mathematical description of Gaussian Splatting in natural color scenes (Equation (1)). In CT imaging, the contribution of a voxel to the final projection does not suffer from attenuation due to occlusion by the following voxels, but is instead Beer-Lambert law. Therefore, equation (1) no longer holds. This makes it difficult to highlight the motivation of this paper in terms of algorithm details.

2.  Theoretical contributions appear to be minor. (Formula (3) is the theoretical contributions of this paper). This makes this paper more suitable for delivery to places that are more concerned with simulation applications.

3. The paper needs to compare with Physics-based Monte-Carlo simulations [2, 15, 3 , 4 ]. Although these methods are time-consuming, they guarantee the quality of reprojection. The paper needs to clearly point out the difference in image quality with Physics-based Monte-Carlo simulations. The ground truth(GT) needs to select the real projection obtained by the CT machine. So the comparison table will be：
______________________________________________________________________________________
                         |    Physics-based Monte-Carlo simulations     |     3DGS   |  X-Gaussian  |   DDGS
_____________________________________________________________________________________
PSNR with GT |  
______________________________________________________________________________________
SSIM  with GT | 
______________________________________________________________________________________
questions: questions: 1. what is the means of high-dimensional residual contribution？Is there a formal description? This concept is not explained in the full text.

2. Before use the method, it need first get the 3D CT volume. The reconstruction is based on the projection obtained by the CT machine. However, there are errors and noise in the 3D CT volume (from the reconstruction of the projection noise) and also from the reconstruction errors caused by the polychromatic X-rays in medical CT machine. The paper is equivalent to reprojecting the 3D CT volume using gaussian splatting. How do the work ensure the reprojection process does not amplify this error from the 3D CT volume?
limitations: limitations: 1. If this method is used in clinical practice, what problems will it cause without considering the time cost? For example, the appearance of artifacts, because once artifacts appear, it will seriously interfere with the doctor's clinical operation.

Comments: title: Response to Reviewer ZGY5
comment: Thank you once again for your thoughtful review and the time that you have dedicated to our work. We truly appreciate your recognition and support of our contributions to DRR rendering. We are especially pleased that our responses not only clarified the points that you raised, but also addressed the concerns of other reviewers from your perspective. We will carefully incorporate your feedback—such as CT visualizations, error maps, and insights on the convergence boost provided by our novel initialization—into the final version of the paper, to further strengthen it.
We would like to thank the reviewer again for their thoughtful review and for the time dedicated to our submission. We appreciate your recognition of our work and acknowledgement of our contributions ("_innovative approach_", "_significant empirical improvements_", "_detailed methodological framework_"). We will account for your insightful remarks in our final version (additional results, further details on rendering process, and insight into the benefits of GS representation over voxel one for DRR).
Thank you for the detailed response, which addresses my concerns. I have also reviewed the comments from other reviewers. I believe this paper makes a significant contribution to X-ray imaging. Therefore, I am raising my score from 5 to 6.
After reading the author's response and the other reviews, I would like to keep my initial score.
Dear `ZGY5`,

Thank you for your message.

We believe that we have addressed your comments, as well as those from other reviewers, in our responses. However, we are concerned about the blank “rejection” decision and would appreciate if more **detailed** and **concrete** reasons for this outcome could be provided. This would greatly help us understand the reviewer's perspective and further improve our work.

Sincerely,

The authors
After reading the author's response and the other reviews, I would like to keep my initial score.
We appreciate `g5Vp`'s feedback and are glad to hear that "_the author's answer solved some of my doubts._" Given that some of the previously mentioned weaknesses have been addressed, may we respectfully ask `g5Vp` to kindly elaborate on their decision to nevertheless opt for _borderline reject_? We are eager to address any remaining concerns and to further improve our submission based on the feedback.

In our view, we have responded to the key concerns raised by `g5Vp`, specifically:

- **Weakness #1 = theoretical misunderstanding by `g5Vp`**. We hope that we have clarified the application of the Beer-Lambert Law to correct the reviewer's understanding.

- **Weakness #2 = significant contributions compared to SOTA**. We acknowledge that the novelty and impact of a paper are subjective, so we have highlighted the theoretical and empirical significance of our contributions relative to the SOTA  [7, 11, 12].

- **Weakness #3 = additional results provided**. We exceeded the standard analytical DRR evaluation [5, 7, 11, 12, 33] by comparing our solution to real projections, as requested by the reviewer.

We have also thoroughly addressed the questions of other reviewers, particularly those raised by `ZGY5` regarding the SOTA status of [7, 11, 12] and the paper’s organization. We note that overall, the reviewers' concerns/questions do not have major overlaps, as opposed to their positive feedback ("novelty" [`Zg8U`, `YFhr`, `ZGY5`], "substantial improvement on target tasks" [`all`], "clear motivation/methodology" [`all`], etc.).

Finally, we would like to emphasize that we submitted this paper to **NeurIPS's "_Machine learning for healthcare_" primary area**. We believe our solution is **well-suited to this application area**, given its ML contributions and focus on clinical tasks. Thus, we respectfully disagree with the reviewer's suggestion that our submission might be better suited to a different venue.

We kindly ask `g5Vp` to consider the arguments above; and if `g5Vp` still believes that our submission is not appropriate for NeurIPS (_Machine learning for healthcare_), we would be thankful for an explanation.

Sincerely,

The Authors
Thanks, I think the author's answer solved some of my doubts, but I decided to keep my score.