Title: R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Gaussian Splatting

3D Reconstruction

CT Reconstruction

Tomographic Reconstruction
Abstract: 3D Gaussian splatting (3DGS) has shown promising results in image rendering and surface reconstruction. However, its potential in volumetric reconstruction tasks, such as X-ray computed tomography, remains under-explored. This paper introduces R$^2$-Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction. By carefully deriving X-ray rasterization functions, we discover a previously unknown \emph{integration bias} in the standard 3DGS formulation, which hampers accurate volume retrieval. To address this issue, we propose a novel rectification technique via refactoring the projection from 3D to 2D Gaussians. Our new method presents three key innovations: (1) introducing tailored Gaussian kernels, (2) extending rasterization to X-ray imaging, and (3) developing a CUDA-based differentiable voxelizer. Experiments on synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art approaches in accuracy and efficiency. Crucially, it delivers high-quality results in 4 minutes, which is 12$\times$ faster than NeRF-based methods and on par with traditional algorithms.

R 2 -Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction

Abstract

3D Gaussian splatting (3DGS) has shown promising results in image rendering and surface reconstruction. However, its potential in volumetric reconstruction tasks, such as X-ray computed tomography, remains under-explored. This paper introduces R 2 -Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction. By carefully deriving X-ray rasterization functions, we discover a previously unknown integration bias in the standard 3DGS formulation, which hampers accurate volume retrieval. To address this issue, we propose a novel rectification technique via refactoring the projection from 3D to 2D Gaussians. Our new method presents three key innovations: (1) introducing tailored Gaussian kernels, (2) extending rasterization to X-ray imaging, and (3) developing a CUDA-based differentiable voxelizer. Experiments on synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art approaches in accuracy and efficiency. Crucially, it delivers high-quality results in 4 minutes, which is 12 × faster than NeRF-based methods and on par with traditional algorithms. Code and models are available on the project page https://github.com/Ruyi-Zha/r2_gaussian .

1 Introduction

Computed tomography (CT) is an essential imaging technique for noninvasively examining the internal structure of objects. Most CT systems use X-rays as the imaging source thanks to their ability to penetrate solid substances [20]. During a CT scan, an X-ray machine captures multi-angle 2D projections that measure ray attenuation through the material. As the core of CT, tomographic reconstruction aims to recover the 3D density field of the object from its projections. This task is challenging in two aspects. Firstly, the harmful X-ray radiation limits the acquisition of sufficient and noise-free projections, making reconstruction a complex and ill-posed problem. Secondly, time-sensitive applications like medical diagnosis require algorithms to deliver results promptly.

Existing tomography methods suffer from either suboptimal reconstruction quality or slow processing speed. Traditional CT algorithms [13, 2, 55] deliver results in minutes but induce serious artifacts. Supervised learning-based approaches [32, 33, 10, 35] achieve promising outcomes by learning semantic priors but struggle with out-of-distribution objects. Recently, neural radiance fields (NeRF) [43] have been applied to tomography and perform well in per-case reconstruction [67, 66, 48, 6, 54]. However, they are very time-consuming ( > 30 minutes) because a huge amount of points have to be sampled for volume rendering.

Chest (50-view)

Ours (iter=30k)

35.01 dB

, 10 min

Ours (iter=10k)

34.56 dB,

3 min

SAX-NeRF

34.45 dB, 792 min

NAF

33.99 dB, 32 min

IntraTomo

33.19 dB, 138 min

Recently, 3D Gaussian splatting (3DGS) [23] has outperformed NeRF in both quality and efficiency for view synthesis [64, 38, 31] and surface reconstruction [16, 18, 65]. However, attempts to apply the 3DGS technique to volumetric reconstruction tasks, such as X-ray tomography, are limited and ineffective. Some concurrent works [7, 14] empirically modify 3DGS for X-ray view synthesis, but they treat it solely as a data augmentation tool for traditional tomography algorithms. To date, there is no 3DGS-based method for direct CT reconstruction.

In this paper, we reveal an inherent integration bias in 3DGS. This bias, despite having a negligible impact on image rendering, critically hampers volumetric reconstruction. To be more specific, we will show in Sec. 4.2.1 that the standard 3DGS overlooks a covariance-related scaling factor when splatting a 3D Gaussian kernel onto the 2D image plane. This formulation leads to inconsistent volumetric properties queried from different views. Besides the integration bias, there are other challenges in applying 3DGS to tomography, such as the difference between natural light and X-ray imaging and the lack of an effective technique to query volumes from kernels.

We propose R 2 -Gaussian (Rectified Radiative Gaussians) to extend 3DGS to sparse-view tomographic reconstruction. R 2 -Gaussian achieves a bias-free training pipeline with three significant improvements. Firstly , we introduce a novel radiative Gaussian kernel, which acts as a local density field parameterized by central density, position, and covariance. We initialize Gaussian parameters using the analytical method FDK [13] and optimize them with photometric losses. Secondly , we rectify the 3DGS rasterizer to support X-ray imaging. This is achieved by deriving new X-ray rendering functions and correcting the integration bias for accurate density retrieval. Thirdly , we develop a CUDA-based differentiable voxelizer, which not only extracts 3D volumes from Gaussians but also enables voxel-based regularization during training. We evaluate R 2 -Gaussian on both synthetic and real-world datasets. Extensive experiments demonstrate that our method surpasses state-of-the-art (SOTA) methods within 4 minutes, which is 12 × faster than the most efficient NeRF-based solution, NAF [67] and comparable to traditional algorithms. It converges to optimal results in 15 minutes, improving PSNR by 0.6 dB compared to SOTA methods. A visual comparison is shown in Fig. 1.

Our contributions can be summarized as follows: (1) We discover a previously unknown integration bias in 3DGS that impedes volumetric reconstruction. (2) We propose the first 3DGS-based tomography framework by introducing new kernels, extending rasterization to X-ray imaging, and developing a differentiable voxelizer. (3) Our method significantly outperforms state-of-the-art methods in both reconstruction quality and training speed, highlighting its practical value.

2 Related work

Tomographic reconstruction Computed tomography (CT) is widely used for non-intrusive inspection in medicine [17, 22], biology [12, 39, 24], and industry [11]. Conventional fan-beam CT produces a 3D volume by reconstructing each slice from 1D projection arrays. Recently, the cone-beam scanner has become popular for its fast scanning and high resolution [52], leading to the demand for 3D tomography, i.e., recovering the volume directly from 2D projection images. Our work focuses on 3D sparse-view reconstruction where less than a hundred projections are captured to reduce radiation exposure. Traditional algorithms are mainly grouped into analytical and iterative methods. Analytical

methods like filtered back projection (FBP) and its 3D variant FDK [13] produce results instantly ( < 1 second) by solving the Radon transform and its inverse [46]. However, they introduce serious streak artifacts in sparse-view scenarios. Iterative methods [2, 55, 40, 51] formulate tomography as a maximum-a-posteriori problem and iteratively minimize the energy function with regularizations. They successfully suppress artifacts but take longer time ( < 10 minutes) and lose structure details. Deep learning methods can be categorized as supervised and self-supervised families. Supervised methods learn semantic priors from CT datasets. They then use the trained networks to inpaint projections [3, 15], denoise volumes [10, 28, 35, 37] or directly output results [19, 63, 1, 32, 33]. Supervised learning methods perform well in cases similar to training sets but suffer from poor generation ability when applied to unseen data. To overcome this limitation, some studies [67, 66, 48, 6, 54] handle tomography in a self-supervised learning fashion. Inspired by NeRF [43], they model the density field with coordinate-based networks and optimize them with photometric losses. Although NeRF-based methods excel in per-case reconstruction, they are time-consuming ( > 30 minutes) due to the extensive point sampling in volume rendering. Our work can be put into the self-supervised learning family, but it greatly accelerates the training process and improves reconstruction quality.

3DGS 3D Gaussian splatting [23] outperforms NeRF in speed by leveraging highly parallelized rasterization for image rendering. 3DGS represents objects with a set of trainable Gaussian-shaped primitives. It has achieved great success in RGB tasks, including surface reconstruction [16, 18, 65], dynamic scene modeling [60, 34, 61], human avatar [36, 30, 27], 3D generation [57, 62, 9], etc. Some concurrent works have extended 3DGS to X-ray imaging. X-Gaussian [7] modify 3DGS to synthesize novel-view X-ray projections. Gao et al. [14] improve X-Gaussian by considering complex noise-inducing physical effects. While they produce plausible 2D X-ray projections, they cannot directly extract 3D density volumes from trained Gaussians. Instead, they first augment projections with 3DGS, and then use traditional algorithms such as FDK for CT reconstruction, which is neither efficient nor effective. Li et al. [29] represent the density field with customized Gaussian kernels, but they replace the efficient rasterization with existing CT simulators. In comparison, our work can both rasterize X-ray projections and voxelize density volumes from Gaussians.

3 Preliminary

3.1 X-ray imaging

A projection I ∈ R H × W measures ray attenuation through the material as shown in Fig. 2. For an Xray r ( t ) = o + t d ∈ R 3 with initial intensity I 0 and path bounds t n and t f , the corresponding raw pixel value I ′ ( r ) is given with the Beer-Lambert Law [20] by: I ′ ( r ) = I 0 exp( -∫ t f t n σ ( r ( t )) dt ) . Here, σ ( x ) is the isotropic density (or attenuation coefficient in physics) at position x ∈ R 3 . Tomography typically transforms raw data to the logarithmic space for computational simplicity, i.e., where each pixel value I ( r ) represents the density integral along the ray path. Except otherwise specified, we use the logarithmic projections as inputs. The goal of tomographic reconstruction is to esti-

Circular

Scanning

Source

Object

σ

(

x

)

r

(

t

)

t

n

t

f

Projection

I

I

(

r

)

mate the 3D distribution of σ ( x ) , output as a discrete volume, with X-ray projections { I i } i =1 , ··· ,N captured from N different angles. Note that real-world projections contain minor anisotropic physical effects such as Compton scattering. Following previous works [13, 2, 55, 67], we do not explicitly model them but treat them as noise during the reconstruction.

3.2 3D Gaussian splatting

3D Gaussian splatting [23] models the scene with a set of 3D Gaussian kernels G 3 = { G 3 i } i =1 , ··· ,M , each parameterized by position, covariance, color, and opacity. A rasterizer R renders an RGB image

<latexi sh

1_b

64="PEH

VM3mO

Q

IA

S7

0+XLk

>

B9

c

C

gN

F

8RW

U

v

p

J

uYn

w

2Z

5

/o

K

D

d

z

q

y

r

G

T

f

j

<latexi sh

1_b

64="

wT

MGfD

K

mN

/

E0o

7

9C

>A

B8X

c

V

FL

z

+

p

u

HQ

R

W

Z

Y

j

Un

r

yI

2

J

5

q

v

k

O

d

3

P

<latexi sh

1_b

64="cgNXU/O

zK

W

E

8

YPuq

>A

B7H

V

S

J3

r

9L

0

o

F

m

p

Q

+

T

D

I

G

Z

j

w

y

M

C

nf

d

R

5

2

k

v

<latexi sh

1_b

64="

CzM

fK0EAG

DIkjm

VUTo

9

>

B+3

c

LSg

F

2v

7d

QN

O

y

n

J

/

R

XYW

u

r

w

q

8

P

5

<latexi sh

1_b

64="0D/

HFC

odPZ

Q

>A

B7

c

NS8

EJ3

q

9L

m

p

+

z

T

I

G

j

Y

w

y

M

X

O

W

nf

g

K

5

2

k

<latexi sh

1_b

64="g

2

EWcFvZ

5LH

+u kV7

M

>A

B9

D

S

N

3

r

J

U

KR

mX

j

Pp

0

I

/Q

8

o

G

d

Y

y

C

f

q

O

T

<latexi sh

1_b

64="WPyf3EMAo/

qK7SOD5

H

8

k

>

CG

cdVBL

w

z

r

F

g

ZN

u

2

9

T

J

m

v

X

Q

j

Y

+

p

U

n

0

R

I

<latexi sh

1_b

64="j8U+P

g

yVE7

dQH

B

Z

M

>A

CG3

c

LSw

z

r

W

F

N

2

9

m

D

J

/v

X

u

o

O

Y

5

f

k

pK

n

0

q

T

I

R

<latexi sh

1_b

64="HU

7

IuQBw dyEC3

Opv

+

f

>A

cX

V

L

FN

M0

Z

5j

q

k

J

z

g/

2

Y

R

W

o

9

Sm

D

n

r

G

8

P

K

T

<latexi sh

1_b

64="

R

NU9

VdKX

Q

q

D

3

zk

>A

CH

c

Z

LS

FIYn mu

2

p

o

g

EMJ

O

T

5

+j w8

/

y

r

7v

G

W

B

f

0

P

<latexi sh

1_b

64="fJm vLq

S

zAXd7

Cw

/

I

>

B

H

c

VD

N

F

3U

2

u

o

g

R

G5

QO

E

T

0Z

k

9

r

y

p

n

8W

K

Y

P

M

j

+

<latexi sh

1_b

64="

zm5/uPAvH

p

FISj

N30

>

B9X

c

V

wM

2r qn

E

yC

7J

o

Y

Z

D

Q

W

k

R

+

Ug

8

T

dL

O

G

K

f

<latexi sh

1_b

64="2nEUPTw/XyKH

AW

+0

Z

7jfk

>

B

3

c

V

NS

M

J

r

q

9

Ivg

R

L

Q

m

C

z

u5

8

Y

Od

o

G

p

F

D

G

3

i

(

x

) =

ρ

i

·

e

-

1

2

(

x

-

p

i

)

/latticetop

Σ

-

1

i

(

x

-

p

i

)

ρ

i

p

i

Σ

i

- Central Density

- Position

- Covariance

G

3

1

G

3

2

σ

(

x

) =

M

∑

i

=1

G

3

i

(

x

)

Density Field Modelling

Training

Real Projection

Dimension

Voxelization

Initial Gaussians

Optimized Gaussians

Final Density Volume

I rgb ∈ R H × W × 3 from these Gaussians, formulated as where T , P , and C are the transformation, projection, and composition modules, respectively. First, T transforms the 3D Gaussians into the ray space, aligning viewing rays with the coordinate axis to enhance computational efficiency. The transformed 3D Gaussians are then projected onto the image plane: G 2 = P ( G 3 ) . The projected 2D Gaussian retains the same opacity and color as its 3D counterpart but omits the third row and column of position and covariance. An RGB image is then rendered by compositing these 2D Gaussians using alpha-blending [45]: I rgb = C ( G 2 ) . The rasterizer is differentiable, allowing for the optimization of kernel parameters using photometric losses. 3DGS initializes sparse Gaussians with structure-from-motion (SfM) points [53]. During training, an adaptive control strategy dynamically densifies Gaussians to improve scene representation.

4 Method

In this section, we first introduce radiative Gaussian as a novel object representation in Sec. 4.1. Next, we adapt 3DGS to tomography in Sec. 4.2. Specifically, we derive new rasterization functions and analyze the integration bias of standard 3DGS in Sec. 4.2.1. We further develop a differentiable voxelizer for volume retrieval in Sec. 4.2.2. The optimization strategy is elaborated in Sec. 4.2.3.

4.1 Representing objects with radiative Gaussians

As shown in Fig. 3, we represent the target object with a group of learnable 3D kernels G 3 = { G 3 i } i =1 , ··· ,M that we term as radiative Gaussians. Each kernel G 3 i defines a local Gaussian-shaped density field, i.e.,

where ρ i , p i ∈ R 3 and Σ i ∈ R 3 × 3 are learnable parameters representing central density, position and covariance, respectively. For optimization purposes, we follow [23] to further decompose the covariance matrix Σ i into the rotation matrix R i and scale matrix S i : Σ i = R i S i S ⊤ i R ⊤ i . The overall density at position x ∈ R 3 is then computed by summing the density contribution of kernels:

Compared with standard 3DGS, our kernel formulation removes view-dependent color because X-ray attenuation depends only on isotropic density, as shown in Eq. (1). More importantly, we define the density query function (Eq. (4)) for radiative Gaussians, making them useful for both 2D image rendering and 3D volume reconstruction. In contrast, the opacity in 3DGS is empirically designed for RGB rendering, leading to challenges when extracting 3D models such as meshes from Gaussians [16, 8, 65]. Concurrent work [29] also explores kernel-based representation but uses simplified isotropic Gaussians. Our work employs a general Gaussian distribution, offering more flexibility and precision in modeling complex structures.

<latexi sh

1_b

64="P5

f

Z08

3oQ

RmM

K

uyc

>A

B7n

V

NSw

EJ2

X

+

T

C

U9k

j

WvH

YD

g

z

/

F

d

p

LG

q

O

r

I

<latexi sh

1_b

64="rPu vfj9UMRF

EW

0

Z

qA

>

B

n

c

V

NS8

J3

X

O

Kk

Hg

+w

p

2

YTd

C

I

D

5

Q

/

7

z

o

Ly

<latexi sh

1_b

64="9EHLTd2MuDXp c7

0j

QI

>A

B

n

V

N

8

J3

W

O

K

R

r

g

F+w

v

Y

Cf

5

P

/

z

G

q

U

o

Z

y

<latexi sh

1_b

64="Z/0

O

M

EU

+

n

uqr

Q8o

>A

B

c

V

NS

J3

W

X

KkmR

Hg

F

w

v

p

2

YTd

Cf

I

D

9

5

P

j

7

z

L

G

y

<latexi sh

1_b

64="9O8nCkRTW

wE

Y3Z5

D

/

Um

I

>A

c

V

7

M

FL0pr

J

S

o

g

B

q

+

2

G

Q

uf

d

z

v

y

N

j

PK

X

H

<latexi sh

1_b

64="z

q

rpL

X

0AEv

2U/

>

B

3

c

V

N

W

O

Y

RI9

jw

GC

M

y7d

D

5

+

P

g

m

k

o

Z

n

u

F

H

f

<latexi sh

1_b

64="Q

G

730mROuknNW8

J

Do

C

>A

B

c

V

S

E

Ur q/

9L

K

I

F

2

Y

g

+

z

H

X

w

v

p

5

Z

d

j

f

M

y

T

P

<latexi sh

1_b

64="O7S

w

H9

v

m

Q

>A

B8n

V

N

EJ3

W

K

R

M

+

F

p

2

yY

f

D

P

/

I

Z

u

z

o

0

q

5

L

U

G

<latexi sh

1_b

64="P3K

V

z

Y

u

mfnv

>A

B7H

c

NS8

EJ

Ur

q

9L

R

Xj

D

d

yO

0

w

Z

k

o

T

<latexi sh

1_b

64="

v9X

u+kLnJq2/j

w

I8pZU3

>A

B

H

c

VD

SgN

EOy

r

f

Y

o

7

M

PG

T

W

m

Q

F

R

5

z

0

K

C

d

(a) Training Pipeline

FDK Volume

Initialization

Operation Flow

Gradient Flow

Dimension

L1

DSSIM

Real

X-ray Rasterizer

Tiny Volume

Regularization

TV(                                    )

Radiative Gaussians

Adaptive Control

Density Voxelizer

Rendered

(c) Density Voxelizer (§4.2.2)

3D Tile

Voxel

Radiative Gaussians

Density Volume

(d) Adaptive Control (§4.2.3)

(b) X-ray Rasterizer (§4.2.1)

Transformation

World Space

≈

x

2

x

0

x

1

∫

dx

2

Projection

Composition

∑

(

)

2D Gaussians

Rendered

ρ

i

(2

π

|

˜

Σ

i

|

/

|

ˆ

Σ

i

|

)

1

/

2

= ˆ

ρ

i

Clone

Split

Prune

O

Initialization 3DGS initializes Gaussians with SfM points, which is not applicable to volumetric tomography. Instead, we initialize our radiative Gaussians using preliminary results obtained from the analytical method. Specifically, we use FDK [13] to reconstruct a low-quality volume in less than 1 second. We then exclude empty spaces with a density threshold τ and randomly sample M points as kernel positions. Following [23], we set the scales of Gaussians as the nearest neighbor distances and assume no rotation. The central densities are queried from the FDK volume. We empirically scale down the queried densities with k to compensate for the overlay between kernels.

4.2 Training radiative Gaussians

Our training pipeline is shown in Fig. 4. Radiative Gaussians are first initialized from an FDK volume. We then rasterize projections for photometric losses and voxelize tiny density volumes for 3D regularization. Adaptive control is used to densify Gaussians for better representation. After training, we voxelize density volumes of the target size for evaluation.

4.2.1 X-ray rasterization

This section focuses on the theoretical derivation of X-ray rasterization R . As discussed in Sec. 3.1, the pixel value of a projection is the integral of density along the corresponding ray path. We substitute Eq. (4) into Eq. (1), yielding

where I r ( r ) is the rendered pixel value. This implies that we can individually integrate each 3D Gaussian to rasterize an X-ray projection. Note that t n and t f in Eq. (1) are neglected because we assume all Gaussians are bounded inside the target space.

Transformation Since a cone-beam X-ray scanner can be modeled similarly to a pinhole camera, we follow [69] to transfer Gaussians from the world space to the ray space. In ray space, the

viewing rays are parallel to the third coordinate axis, facilitating analytical integration. Due to the non-Cartesian nature of ray space, we employ the local affine transformation to Eq. (5), yielding

where ˜ x = [ x 0 , x 1 , x 2 ] ⊤ is a point in ray space, ˜ p i ∈ R 3 is the new Gaussian position obtained through projective mapping ϕ , and ˜ Σ i ∈ R 3 × 3 is the new Gaussian covariance controlled by local approximation matrix J i and viewing transformation matrix W . Refer to Appendix A for determining ϕ , J i , and W from scanner parameters.

Projection and composition A good property of normalized 3D Gaussian distribution is that its integral along one coordinate axis yields a normalized 2D Gaussian distribution. Substitute Eq. (3) into Eq. (6) and we have

where ˆ x ∈ R 2 , ˆ p ∈ R 2 , ˆ Σ ∈ R 2 × 2 are obtained by dropping the third rows and columns of their counterparts ˜ x , ˜ p , and ˜ Σ , respectively. Eq. (7) shows that an X-ray projection can be rendered by simply summing 2D Gaussians instead of alpha-compositing them in natural light imaging.

Integration bias During the projection, a key difference between our 2D Gaussian and the original one in 3DGS is the central density (opacity) ˆ ρ i . As shown in Eq. (7), we scale the density with a covariance-related factor µ i = (2 π | ˜ Σ i | / | ˆ Σ i | ) 1 / 2 : ˆ ρ i = µ i ρ i , while 3DGS does not. This implies that 3DGS, in fact, learns an integrated density in the 2D image plane rather than the actual one in 3D space. This integration bias, though having a negligible impact on imaging rendering, leads to significant inconsistency in density retrieval. We demonstrate the inconsistency with a simplified 2D-to-1D projection in Fig. 5. When attempting to recover the central density ρ in 3D space with ρ i = ˆ ρ i /µ j , we find different views ( µ j ) lead to different results. This violates the isotropic nature of ρ i , preventing us from determining the correct value. In contrast, our method assigns the actual 3D density to the kernel and forwardly computes the 2D projection, thus fundamentally solving the issue. While conceptually simple, implementing our idea requires substantial engineering efforts, including reprogramming all backpropagation routines in CUDA.

4.2.2 Density voxelization

We develop a voxelizer V to efficiently query a density volume V ∈ R X × Y × Z from radiative Gaussians: V = V ( G 3 ) . Inspired by voxelizers used in RGB tasks [57], our voxelizer first partitions the target space into multiple 8 × 8 × 8 3D tiles. It then culls Gaussians, retaining those with a 99% confidence of intersecting the tile. In each 3D tile, voxel values are parallelly computed by summing the contributions of nearby kernels with Eq. (4). We implement the voxelizer and its backpropagation in CUDA, making it differentiable for optimization. This design not only accelerates the query process ( > 100 FPS) but also allows us to regularize radiative Gaussians with 3D priors.

ρ

= ˆ

ρ/µ

1

or

ρ

= ˆ

ρ/µ

2

?

µ

1

µ

2

ρ

ˆ

ρ

ˆ

ρ

<latexi sh

1_b

64="2wF7

R

I

PMd/LV

jq

H

Bzk

>A

Cf3

c

Z

N

9

G

8D

Ky

o

rE

SY

XJ

O

U

0m

n

+

Q

5g

W

v

p

<latexi sh

1_b

64="

92NV03o

W5jz

A

f

ZPu

>

B

c

C7Sg

FL

rM8

U

X

TR

HJEmYn

k

wp

D

/Id

Q

v

G

q

y

O

+

K

<latexi sh

1_b

64="Kpwzy

7H9

MY0

TV2grNP

J

>A

CGX

c

ZDLS

FI

n

mu

E

/

5

f

Q

B

R

O

d

o

j

+

8

U

3

q

W

v

k

<latexi sh

1_b

64="c

P9VOM3y

Dkngz8

w

UH

R

>A

B

X

LS

N

FJ

q

C

m

2Y

/

5

0

I

7

WGjr

Qu

+

v

o

d

E

f

<latexi sh

1_b

64="

Uqyfm9

C

+A

u

n8

FR

Y

>

3

c

ZDLSgM

I

z

V

r

d

E

T

BX

KJk0

5

vo

Q

H

W

7

w

/

O

P

2

p

j

G

N

<latexi sh

1_b

64="c

P9VOM3y

Dkngz8

w

UH

R

>A

B

X

LS

N

FJ

q

C

m

2Y

/

5

0

I

7

WGjr

Qu

+

v

o

d

E

f

4.2.3 Optimization

We optimize radiative Gaussians using stochastic gradient descent. Besides photometric L1 loss L 1 and D-SSIM loss L ssim [59], we further incorporate a 3D total variation (TV) regularization [49] L tv as a homogeneity prior for tomography. At each training iteration, we randomly query a tiny density volume V tv ∈ R D × D × D (same spacing as the target output) and minimize its total variation. The overall training loss is defined as:

where I r , I m , λ ssim and λ tv are rendered projection, measured projection, D-SSIM weight, and TV weight, respectively. Adaptive control is employed during training to enhance object representation. We remove empty Gaussians and densify (clone or split) those with large loss gradients. Considering objects such as human organs have extensive homogeneous areas, we do not prune large Gaussians. As for densification, we halve the densities of both original and replicated Gaussians. This strategy mitigates the sudden performance drop caused by new Gaussians and hence stabilizes training.

5 Experiments

5.1 Experimental settings

Dataset We conduct experiments on both synthetic and real-world datasets. For the synthetic dataset, we collect 15 real CT volumes, ranging from organisms to artificial objects. We then use the tomography toolbox TIGRE [5] to synthesize X-ray projections and add Compton scatter and electric noise. For real-world experiments, we use three cases from the FIPS dataset [56], each with 721 real projections. Since ground truth volumes are unavailable, we use FDK [13] to create pseudo-ground truth using all views and then subsample views for sparse-view experiments. We set 75, 50, and 25 views for both synthetic and real-world data as three sparse-view scenarios. Refer to Appendix B for more details of datasets.

Implementation details Our R 2 -Gaussian is implemented in PyTorch [44] and CUDA [50], and trained with the Adam optimizer [25] for 30k iterations. Learning rates for position, density, scale, and rotation are initially set as 0.0002, 0.01, 0.005, and 0.001, respectively, and exponentially to 0.1 of their initial values. Loss weights are λ ssim = 0 . 25 and λ tv = 0 . 05 . We initialize M = 50 k Gaussians with a density threshold τ = 0 . 05 and scaling term k = 0 . 15 . The TV volume size is

Ground Truth

Ours

SAX-NeRF

NAF

IntraTomo

ASD-POCS

SART

FDK

Pepper

25.68

34.03

36.92

33.95

37.18

36.87

39.45

75-view

Chest

24.37

31.86

32.37

33.18

33.99

34.45

35.01

50-view

30.19

29.51

32.05

32.70

32.74

35.25

25-view

Pine

30.48

32.41

33.50

38.73

39.35

36.96

40.38

50-view

5.2 Results and evaluation

For fairness, we do not compare methods that require external training data but focus on those that solely use 2D projections of arbitrary objects. We compare R 2 -Gaussian with three traditional methods (FDK [13], SART [2], ASD-POCS [55]) and three SOTA NeRF-based methods (IntraTomo [66], NAF [67], SAX-NeRF [6]). Tab. 1 reports the quantitative results on sparse-view tomography. Note that we do not report the running time for FDK as it is instant. R 2 -Gaussian achieves the best performance across all synthetic and most real-world experiments. Specifically, our method delivers a 0.93 dB higher PSNR than SAX-NeRF, on the synthetic dataset, and a 0.95 dB improvement over IntraTomo on the real-world dataset. It is also worth noting that our 50-view results are already on par with the 75-view results of other methods. Regarding efficiency, our method converges to optimal results in 15 minutes, which is 3.7 × faster than the most efficient NeRF-based method, NAF. Surprisingly, it takes less than 4 minutes to surpass other methods, which is even faster than the traditional algorithm SART. Fig. 6 shows the visual comparisons of different methods. FDK and SART introduce streak artifacts, while ASD-POCS and IntraTomo blur structural details. NAF and SAX-NeRF are better than other baseline methods but have salt-and-pepper noise. In comparison, our method successfully recovers sharp details, e.g., ovules of pepper, and maintains good smoothness for homogeneous areas, e.g., muscles in the chest.

5.3 Ablation study

Integration bias To demonstrate the impact of integration bias discussed in Sec. 4.2.1, we develop an X-ray version of 3DGS (X-3DGS) that uses X-ray rendering while retaining the biased 3D-to-2D Gaussian projection. We use the same voxelizer in Sec. 4.2.2 to extract volumes. Before voxelization, we divide the learned density of each Gaussian by the mean scaling factor µ of all training views. Tab. 2 shows that rectifying integration bias benefits both 2D rendering (+3.15 dB PSNR) and 3D reconstruction (+17.77 dB PSNR). Fig. 7 visualize rendering and reconstruction results. While X-3DGS renders reasonable 2D projections, its reconstruction quality is significantly worse than ours. Besides, there are notable discrepancies in slices queried from different views. The conflicting 2D and 3D performances indicate that X-3DGS, despite fitting images well, does not accurately model the density field. In contrast, our method learns the actual view-independent density, eliminating inconsistencies and ensuring unbiased object representation.

Component analysis We conduct ablation experiments to assess the effect of FDK initialization (Init.), modified adaptive control (AC), and total variation regularization (Reg.) on performance. The baseline model excludes these components and uses randomly generated Gaussians for initialization. Experiments are performed under the 50-view condition, evaluating PSNR, SSIM, training time, and Gaussian count (Gau.). Results are listed in Tab. 3. FDK initialization boosts PSNR by 0 . 9 dB. Adaptive control improves quality but prolongs training due to more Gaussians. TV regularization increases SSIM by reducing artifacts and promoting smoothness. Overall, our full model outperforms the baseline, improving PSNR by 1.51 dB and SSIM by 0.018, with training time under 9 minutes.

Parameter analysis We perform parameter analysis on the number of initialized Gaussians M , TV loss weight λ tv , and TV volume size D . The results are shown in the last three blocks of Tab. 3. R 2 -Gaussian achieves good quality-efficiency balance at 50k initialized Gaussians. A TV loss weight of λ tv = 0 . 05 improves reconstruction, but larger values can lead to degradation. The training time increases with TV volume size while the performance peaks at D = 32 .

Convergence analysis Fig. 8 compares the results of NeRF-based methods and our R 2 -Gaussian method at different iterations. Our method, both with and without FDK initialization, converges significantly faster, displaying sharp details by the 500th iteration when other methods still exhibit artifacts and blurriness. Notably, the FDK initialization offers a rough structure before training, which

30°

90°

150°

Ground Truth

Ours

X-3DGS (150°)

X-3DGS (90°)

X-3DGS (30°)

Ours

X-3DGS

Beetle (50-view)

Reconstructed Volumes

Rendered Projections (30°)

46.38

46.20

34.97

34.72

34.78

42.39

1

↑

Engine (50-view)

1

st

iter

250

th

iter

500

th

iter

750

th

iter

1000

th

iter

Final

Ours

38.90 dB

,

9 min

Ours w/o Init.

38.63 dB, 10 min

SAX-NeRF

36.90 dB, 793 min

NAF

37.12 dB, 32 min

IntraTomo

35.04 dB, 126 min

⋮

⋮

⋮

⋮

⋮

0

further accelerates convergence and enhances reconstruction quality. Finally, our method outperforms others in both performance and efficiency, achieving the highest PSNR of 38.90 dB in 9 minutes.

6 Discussion and conclusion

Discussion R 2 -Gaussian inherits some limitations from 3DGS, such as varying training time across modalities, needle-like artifacts under extremely sparse-view conditions, and suboptimal extrapolation for other tomography tasks. Besides, we have not considered calibration errors regarding the scanned geometry and anisotropic physical effects such as Compton scattering. More details are discussed in Appendix E. Despite these limitations, our method's superior performance and fast speed make it valuable for real-world applications for medical diagnosis and industrial inspection.

Conclusion This paper presents R 2 -Gaussian, a novel 3DGS-based framework for sparse-view tomographic reconstruction. We identify and rectify a previously overlooked integration bias of standard 3DGS, which hinders accurate density retrieval. Furthermore, we enhance 3DGS for tomography by introducing new kernels, devising X-ray rasterization functions, and developing a differentiable voxelizer. Our R 2 -Gaussian surpasses state-of-the-art methods in both reconstruction quality and training speed, demonstrating its potential for real-world applications. Crucially, we speculate that the newly found integration bias may be pervasive across all 3DGS-related research. Consequently, our rectification technique could benefit more tasks beyond computed tomography.

Acknowledgments

The research is funded in part by ARC Discovery Grant (grant ID: DP220100800) of the Australia Research Council.

References

Jonas Adler and Ozan Öktem. Learned primal-dual reconstruction. IEEE transactions on medical imaging , 37(6):1322-1332, 2018.

Anders H Andersen and Avinash C Kak. Simultaneous algebraic reconstruction technique (sart): a superior implementation of the art algorithm. Ultrasonic imaging , 6(1):81-94, 1984.

Rushil Anirudh, Hyojin Kim, Jayaraman J Thiagarajan, K Aditya Mohan, Kyle Champley, and Timo Bremer. Lose the views: Limited angle ct reconstruction via implicit sinogram completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6343-6352, 2018.

Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics , 38(2):915-931, 2011.

Ander Biguri, Manjit Dosanjh, Steven Hancock, and Manuchehr Soleimani. Tigre: a matlab-gpu toolbox for cbct image reconstruction. Biomedical Physics & Engineering Express , 2(5):055010, 2016.

Yuanhao Cai, Jiahao Wang, Alan Yuille, Zongwei Zhou, and Angtian Wang. Structure-aware sparse-view x-ray 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Yuanhao Cai, Yixun Liang, Jiahao Wang, Angtian Wang, Yulun Zhang, Xiaokang Yang, Zongwei Zhou, and Alan Yuille. Radiative gaussian splatting for efficient x-ray novel view synthesis. In European Conference on Computer Vision , pages 283-299. Springer, 2025.

Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846 , 2023.

Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Hyungjin Chung, Dohoon Ryu, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22542-22551, 2023.

Leonardo De Chiffre, Simone Carmignato, J-P Kruth, Robert Schmitt, and Albert Weckenmann. Industrial applications of computed tomography. CIRP annals , 63(2):655-677, 2014.

Philip CJ Donoghue, Stefan Bengtson, Xi-ping Dong, Neil J Gostling, Therese Huldtgren, John A Cunningham, Chongyu Yin, Zhao Yue, Fan Peng, and Marco Stampanoni. Synchrotron x-ray tomographic microscopy of fossil embryos. Nature , 442(7103):680-683, 2006.

Lee A Feldkamp, Lloyd C Davis, and James W Kress. Practical cone-beam algorithm. Josa a , 1 (6):612-619, 1984.

Zhongpai Gao, Benjamin Planche, Meng Zheng, Xiao Chen, Terrence Chen, and Ziyan Wu. Ddgs-ct: Direction-disentangled gaussian splatting for realistic volume rendering. arXiv preprint arXiv:2406.02518 , 2024.

Muhammad Usman Ghani and W Clem Karl. Deep learning-based sinogram completion for low-dose ct. In 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP) , pages 1-5. IEEE, 2018.

Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Godfrey N Hounsfield. Computed medical imaging. Science , 210(4465):22-28, 1980.

Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers . Association for Computing Machinery, 2024. doi: 10.1145/3641519.3657428.

Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE transactions on image processing , 26(9):4509-4522, 2017.

Avinash C Kak and Malcolm Slaney. Principles of computerized tomographic imaging . SIAM, 2001.

Emma Kamutta, Sofia Mäkinen, and Alexander Meaney. Cone-Beam Computed Tomography Dataset of a Seashell, August 2022. URL https://doi.org/10.5281/zenodo.6983008 .

Shigehiko Katsuragawa and Kunio Doi. Computer-aided diagnosis in chest radiography. Computerized Medical Imaging and Graphics , 31(4-5):212-223, 2007.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Timo Kiljunen, Touko Kaasalainen, Anni Suomalainen, and Mika Kortesniemi. Dental cone beam ct: A review. Physica Medica , 31(8):844-860, 2015.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA, 2015.

Pavol Klacansky. Open scivis datasets, December 2017. URL https://klacansky.com/ open-scivis-datasets/ . https://klacansky.com/open-scivis-datasets/ .

Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. HUGS: Human gaussian splatting. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. URL https://arxiv.org/abs/2311.17910 .

Suhyeon Lee, Hyungjin Chung, Minyoung Park, Jonghyuk Park, Wi-Sun Ryu, and Jong Chul Ye. Improving 3d imaging with pre-trained perpendicular 2d diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10710-10720, 2023.

Yingtai Li, Xueming Fu, Shang Zhao, Ruiyang Jin, and S Kevin Zhou. Sparse-view ct reconstruction with 3d gaussian volumetric representation. arXiv preprint arXiv:2312.15676 , 2023.

Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Yiqun Lin, Zhongjin Luo, Wei Zhao, and Xiaomeng Li. Learning deep intensity field for extremely sparse-view cbct reconstruction. In International Conference on Medical Image Computing and ComputerAssisted Intervention , pages 13-23. Springer, 2023.

Yiqun Lin, Jiewen Yang, Hualiang Wang, Xinpeng Ding, Wei Zhao, and Xiaomeng Li. Cˆ 2rv: Crossregional and cross-view learning for sparse-view cbct reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11205-11214, 2024.

Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Jiaming Liu, Rushil Anirudh, Jayaraman J Thiagarajan, Stewart He, K Aditya Mohan, Ulugbek S Kamilov, and Hyojin Kim. Dolce: A model-based probabilistic diffusion framework for limited-angle ct reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10498-10508, 2023.

Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Zhengchun Liu, Tekin Bicer, Rajkumar Kettimuthu, Doga Gursoy, Francesco De Carlo, and Ian Foster. Tomogan: low-dose synchrotron x-ray tomography with generative adversarial networks: discussion. JOSA A , 37(3):422-434, 2020.

Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Vladan Luˇ ci´ c, Friedrich Förster, and Wolfgang Baumeister. Structural studies by electron tomography: from cells to molecules. Annu. Rev. Biochem. , 74:833-865, 2005.

Stephen H Manglos, George M Gagne, Andrzej Krol, F Deaver Thomas, and Rammohan Narayanaswamy. Transmission maximum-likelihood reconstruction with ordered subsets for cone beam ct. Physics in Medicine & Biology , 40(7):1225, 1995.

Alexander Meaney. Cone-Beam Computed Tomography Dataset of a Pine Cone, August 2022. URL https://doi.org/10.5281/zenodo.6985407 .

Alexander Meaney. Cone-beam computed tomography dataset of a walnut, August 2022. URL https: //doi.org/10.5281/zenodo.6986012 .

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. Advances in neural information processing systems , 32, 2019.

Thomas Porter and Tom Duff. Compositing digital images. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques , pages 253-259, 1984.

Johann Radon. On the determination of functions from their integral values along certain manifolds. IEEE transactions on medical imaging , 5(4):170-176, 1986.

Holger Roth, Amal Farag, Evrim B. Turkbey, Le Lu, Jiamin Liu, and Ronald M. Summers. Data from pancreas-ct, 2016. URL https://www.cancerimagingarchive.net/collection/pancreas-ct/ .

Darius Rückert, Yuanhao Wang, Rui Li, Ramzi Idoughi, and Wolfgang Heidrich. Neat: Neural adaptive tomography. ACM Transactions on Graphics (TOG) , 41(4):1-13, 2022.

Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena , 60(1-4):259-268, 1992.

Jason Sanders and Edward Kandrot. CUDA by example: an introduction to general-purpose GPU programming . Addison-Wesley Professional, 2010.

Ken Sauer and Charles Bouman. A local update strategy for iterative reconstruction from projections. IEEE Transactions on Signal Processing , 41(2):534-548, 1993.

William C Scarfe, Allan G Farman, Predag Sukovic, et al. Clinical applications of cone-beam computed tomography in dental practice. Journal-Canadian Dental Association , 72(1):75, 2006.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016.

Liyue Shen, John Pauly, and Lei Xing. Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Emil Y Sidky and Xiaochuan Pan. Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization. Physics in Medicine & Biology , 53(17):4777, 2008.

The Finnish Inverse Problems Society. X-ray tomographic datasets, 2024. URL https://fips.fi/ category/open-datasets/x-ray-tomographic-datasets/ .

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=UyNXMqnN3c .

Pieter Verboven, Bart Dequeker, Jiaqi He, Michiel Pieters, Leroi Pols, Astrid Tempelaere, Leen Van Doorselaer, Hans Van Cauteren, Ujjwal Verma, Hui Xiao, et al. www. x-plant. org-the ct database of plant organs. In 6th Symposium on X-ray Computed Tomography: Inauguration of the KU Leuven XCT Core Facility, Location: Leuven, Belgium , 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Xingde Ying, Heng Guo, Kai Ma, Jian Wu, Zhengxin Weng, and Yefeng Zheng. X2ct-gan: reconstructing ct from biplanar x-rays with generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10619-10628, 2019.

Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient high-quality compact surface reconstruction in unbounded scenes. arXiv preprint arXiv:2404.10772 , 2024.

Guangming Zang, Ramzi Idoughi, Rui Li, Peter Wonka, and Wolfgang Heidrich. Intratomo: self-supervised learning-based tomography via sinogram synthesis and prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1960-1970, 2021.

Ruyi Zha, Yanhao Zhang, and Hongdong Li. Naf: Neural attenuation fields for sparse-view cbct reconstruction. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 442-452. Springer, 2022.

Zheng Zhang, Wenbo Hu, Yixing Lao, Tong He, and Hengshuang Zhao. Pixel-gs: Density control with pixel-aware gradient for 3d gaussian splatting. arXiv preprint arXiv:2403.15530 , 2024.

Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa splatting. IEEE Transactions on Visualization and Computer Graphics , 8(3):223-238, 2002.

A Transformation module in X-ray rasterization

The configuration of a cone beam CT scanner is shown in Fig. 9. The X-ray source and detector plane rotate around the z-axis, resembling a pinhole camera model. Therefore, we can formulate the field-of-view (FOV) of a scanner as

Here, ( D x , D y ) is the physical size of the detector plane, and L SD is the distance between the source and the detector. Following [23], we then use FOVs to determine the projection mapping ϕ .

To get Gaussians in the ray space, we first transfer them from the world space to the scanner space. The scanner space is defined such that its origin is the X-ray source, and its z-axis points to the projection center. The transformation matrix T from the world space to the scanner space is

Here, ϕ is the rotation angle, and L SO is the distance between the source and the object. Next, we apply local approximation on each Gaussian. The Jacobian of the affine approximation J i is the same as Eq. (29) in [69]. Finally, we have the Gaussian in the ray space with new position ˜ p and covariance ˜ Σ i as

z

Source Trajectory

Detector Trajectory

L

SO

θ

D

x

D

y

x

y

Detector

Source

B Details of dataset

Synthetic data We evaluate methods with various modalities, covering major CT applications such as medical diagnosis, biological research, and industrial inspection. The synthetic dataset consists of 15 cases across three categories: human organs (chest, foot, head, jaw, and pancreas), animals and plants (beetle, bonsai, broccoli, kingsnake, and pepper), and artificial objects (backpack, engine, present, teapot, and mount). The chest and pancreas scans are from LIDC-IDRI [4] and Pancreas-CT [47], respectively. Broccoli and pepper are obtained from X-Plant [58], and the rest are from SciVis [26]. Following [67, 6], we preprocess raw data by normalizing densities to [0 , 1] and resizing volumes to 256 × 256 × 256 . We then use the tomography toolbox TIGRE [5] to capture 512 × 512 projections in the range of 0 ◦ ∼ 360 ◦ . We add two types of noise: Gaussian (mean 0, standard deviation 10) as electronic noise of the detector and Poisson (lambda 1e5) as photon scattering noise. All volumes and their projection examples are shown in Fig. 10.

Real-world data We use FIPS [56], a public dataset providing real 2D X-ray projections. FIPS includes three objects (pine [41], seashell [21], and walnut [42]). Each case has 721 projections in the range of 0 ◦ -360 ◦ . We preprocess 2D projections by resizing them to 560 x 560 and normalizing them to [0 , 1] . Since ground truth volumes are unavailable, we use FDK to create pseudo-ground truth with all views and then subsample 75/50/25 views for sparse-view experiments. The size of the target volume is 256 × 256 × 256 .

<latexi sh

1_b

64="PAv zc/

oY

mJkI

jHp

D

S

>

B7

V

9

wN

E

2L

3F

U

C

u

n

g

r

+

W

8M

Z

X

dK

O

R

Q

f

q

G

5

T

y

0

<latexi sh

1_b

64="BOwF2nC

RH

g

K

vpA

S

Zk

>

7

c

V

N

8

EJ3Ur

q

j

W

TyX

0P

Y

M

d

L

u

G

of

/

5

9

+

Q

D

m

I

z

<latexi sh

1_b

64="Lc

8

N

ZjozS

WMAJ

2

m

qY

>

B

H

C7

g

FI

P

u

Gd

Rv

w

k

T

O

y

UV

K

D

f

0

/

n

5

3

r

X

E

Q

p

+

9

<latexi sh

1_b

64="Z

JX

DIEfV7Su og

>A

B

H

c

C

N

F

P

M

q

jGd

RvL

w

k

m

2

T

Ozy8

U

YW

K

0

/

n

5

3

r

Q

p

+

9

<latexi sh

1_b

64="ZXT

j

g0JQWH59

u

2D

/

>A

B7

c

C

S

N

FI

n

y3G

R

kME

Vd

O

Y

m

P

U

q

8

fw

+

z

K

o

L

r

v

p

<latexi sh

1_b

64="

YD5OZAGnEd

8SB

F

KN

>

7X

c

V

L

g

y

r

j

0M

C

u

I

Hw

m

k

T

z

Q

/+

9

P

R

oq

p

v2

3

f

W

J

U

<latexi sh

1_b

64="Z

v

y

2

npu/Kq3

cHk

o

>A

B

VC7SgN

FL0

XzG+

P

JQ

8

E

O

I

w9m

Wd

Y

rR

j

f

T

5

U

M

D

Synthetic data

Artificial Objects

Animals and Plants

Human Organs

Backpack

Beetle

Chest

Engine

Foot

Mount

Broccoli

Head

Present

Kingsnake

Teapot

Pepper

Pancreas

Real-world data

Walnut

Seashell

Pine

Projection (90°)

Projection (45°)

Projection (0°)

Volume

Projection (90°)

Projection (45°)

Projection (0°)

Volume

Projection (90°)

Projection (45°)

Projection (0°)

Volume

C Implementation details of baseline methods

For fairness, we do not compare methods that require external training data but focus on those that solely use 2D projections of arbitrary objects. We run traditional algorithms FDK, SART, and ASDPOCS with GPU-accelerated tomographic toolbox TIGRE [5], and select three SOTA NeRF-based tomography methods. IntraTomo models the density field with a large MLP. NAF accelerates the training process by hash encoding. SAX-NeRF achieves plausible results with a line segment-based transformer. We use the official code of NAF and SAX-NeRF and conduct experiments with default hyperparameters. The IntraTomo implementation is sourced from the NAF repository. The training iterations of NeRF-based methods are set to 150k (default of NAF and SAX-NeRF). All methods are run on a single RTX 3090 GPU.

D More qualitative results

Main results We visualize more reconstruction results in Fig. 11 and Fig. 12. FDK and SART introduce notable streak artifacts, while ASD-POCS and IntraTomo blur structural details. NeRF-based solutions perform better than traditional methods but exhibit salt-and-pepper noise. In comparison, our method successfully recovers sharp details and maintains smoothness in homogeneous areas.

Integration bias We show more qualitative comparisons of X-3DGS and ours in Fig. 13. Our method outperforms X-3DGS in both 2D rendering and 3D reconstruction.

Components and parameters Wevisually compare different components and parameters in Fig. 14. Our newly introduced components improve the reconstruction quality. Our parameter setting also yields the best performance.

Convergence analysis We show the PSNR and SSIM plots in Fig. 15. Our method converges significantly faster than NeRF-based methods and outperforms them in only 3000 iterations.

E More discussion of limitation

Varying time We present the training times for all cases in Fig. 16. The training time varies across cases, primarily due to the different numbers of kernels used. Our method takes more time on objects

Ours

SAX-NeRF

NAF

IntraTomo

Ours

SAX-NeRF

NAF

IntraTomo

75-view

50-view

25-view

GT

Ours

SAX-NeRF

NAF

IntraTomo

33.43

35.24

35.92

35.87

33.18

33.99

34.45

35.01

31.85

30.94

31.54

31.90

941%

041%

041%

941%041%

30.72

32.57

32.80

32.52

30.77

31.65

31.90

31.85

30.29

30.02

30.27

30.26

37.67

40.17

41.01

40.96

37.27

38.90

39.51

40.16

36.98

35.66

36.72

36.32

G

32.87

36.04

36.81

37.31

32.81

35.01

35.37

36.44

31.64

31.49

31.68

33.31

36.71

36.65

37.52

37.72

36.21

35.24

35.71

36.14

33.53

31.87

32.33

33.08

37.57

42.10

42.21

43.32

37.79

41.11

41.39

42.45

37.25

38.99

38.81

40.21

33.08

36.39

36.58

35.57

33.15

35.09

35.13

34.89

32.60

32.59

33.12

32.91

29.43

35.37

35.89

36.46

29.67

33.39

34.11

34.41

28.09

29.32

29.65

29.09

38.48

40.15

40.20

39.92

38.42

39.51

39.56

39.71

38.51

38.20

38.58

38.90

33.95

37.18

36.87

39.45

34.04

35.53

35.54

38.30

33.33

32.68

33.10

34.32

32.63

36.32

36.65

38.83

32.50

35.14

35.28

38.07

32.05

32.70

32.74

35.25

37.52

38.73

38.23

40.08

35.05

37.12

36.90

38.90

37.20

33.96

34.69

34.63

33.90

36.12

36.34

38.27

33.96

35.88

35.58

37.78

32.72

32.72

32.50

35.20

43.37

46.30

45.50

48.06

44.18

45.34

44.68

47.35

44.06

43.05

43.18

45.94

39.98

38.20

38.49

38.82

39.79

36.90

37.87

38.21

40.09

34.44

36.03

36.47

with large homogeneous areas, such as the chest, pancreas, and mount, and less time on those with sparse structures, such as the beetle, backpack, and present.

Needle-like artifacts While our method achieves the highest reconstruction quality, it introduces needle-like artifacts, especially under the 25-view condition (Fig. 17). This suggests that some Gaussians may overfit specific X-rays. Similar artifacts are also observed in 3DGS [68].

Extrapolation ability While this paper focuses on sparse-view CT (SVCT), we also test R 2 -Gaussian on limited-angle CT (LACT), where the scanning range is constrained to less than 180 ◦ . Unlike SVCT, which highlights the interpolation ability of methods, LACT challenges their extrapolation ability, i.e., estimating unseen areas outside the scanning range. We generate 100 projections within ranges of 0 ◦ ∼ 150 ◦ , 0 ◦ ∼ 120 ◦ , and 0 ◦ ∼ 90 ◦ . The quantitative results in Tab. 4 show that

Ground Truth

Ours

SAX-NeRF

NAF

IntraTomo

ASD-POCS

SART

FDK

Walnut

25.13

31.32

33.48

32.17

32.88

30.91

33.34

75-view

Pine

30.48

32.41

33.50

38.73

39.35

36.96

40.38

-view

54

34.07

32.84

39.00

35.04

35.55

38.74

25-view

our method has slightly lower PSNR but higher SSIM than the NeRF-based method NAF. Visualization results in Fig. 18 indicate that our method recovers more details in scanned areas but exhibits blurred artifacts in unseen areas. We attribute this performance drop to the nature of the networks and kernels. Given the gradient of a ray, NeRF updates the entire network while 3DGS individually optimizes intersected kernels. Thus, NeRF has better global awareness and consistency, while 3DGS is more local-oriented and has suboptimal extrapolation ability.

Calibration error In real-world applications, calibration errors can affect reconstruction quality. For example, tomography requires a reference image I 0 in Eq. (1) to represent the illumination pattern without the object. This reference image may have artifacts, such as intensity dropoff towards the image boundaries and other non-uniformities in illumination. Additionally, scanner extrinsics and intrinsics may vary during scanning due to heat expansion and mechanical vibrations. Addressing these practical challenges will be the focus of future work.

Anisotropic effects Following existing CT reconstruction methods, we work under the isotropic assumption of X-ray imaging. However, in the real world, some X-ray transport effects, such as Compton scattering, are anisotropic. We do not explicitly model them but treat them as noises on X-ray projections. This is a necessary simplification for CT reconstruction but may induce inaccuracy for novel-view X-ray synthesis. Readers may refer to [7, 14] for using 3DGS for X-ray view synthesis.

Reconstructed Slices

Rendered Projections (0°)

Ground Truth

Ours

X-3DGS

Ground Truth

Ours

X-3DGS

52.09

52.72

28.68

40.96

75-view

0..

8

48.31

49.60

26.33

34.41

50-view

38.89

53.11

23.92

36.47

25-view

F Broader impacts

Impacts on real-world applications Computed tomography is an essential imaging technique that is widely used in fields including medicine, biology, industry, etc. Our R 2 -Gaussian enjoys superior reconstruction performance and fast convergence speed, making it promising to be implemented in real-world applications such as medical diagnosis and industrial inspection.

Impacts on research community We discover a previously unknown integration bias problem in currently popular 3DGS. we speculate that this problem could be universal across all 3DGS-related works. Therefore, our rectification technique may apply to wider practical domains, not limited to tomography but also other tasks such as magnetic resonance imaging (MRI) reconstruction and volumetric-based surface reconstruction.

Ground Truth

B+TV

B+FDK

Baseline (B)

Ground Truth

𝑀 = 200𝑘

𝑀 = 100𝑘

𝑴=𝟓𝟎𝒌

𝑀 = 10𝑘

𝑀 = 5𝑘

Ground Truth

𝜆

!"

= 0.15

𝜆

!"

= 0.1

𝝀

𝒕𝒗

= 𝟎. 𝟎𝟓

𝜆

!"

= 0.01

𝜆

!"

= 0

Ground Truth

𝐷 = 64

𝐷 = 48

𝑫 = 𝟑𝟐

𝐷 = 16

𝐷 = 8

37.83

39.42

39.30

38.42

40.16

50-view

36.51

37.01

37.15

36.99

37.78

50-view

ae

a6

36.06

36.23

36.43

36.39

36.24

50-view

34.74

34.98

35.05

34.95

34.92

50-view

40.0

0.95

35.0

0.90

0.85

PSNR

(aP)

30.0

sno

Ours (Final)

M

0.80

Ours (Final)

25.0

NAF

NAF (Final)

0.75

NAF

NAF (Final)

SAXNeRF

0.70

SAXNeRF

SAXNeRF (Final)

SAXNeRF (Final)

20.0

IntraTomo

0.65

IntraTomo

IntraTomo (Final)

IntraTomo (Final)

5k

10k

15k

20k

25k

30k

5k

10k

15k

20k

25k

30k

Iteration

Iteration

Time (min)

0

5

10

15

Chest

Foot

Head

Pancreas

Beetle

Broccoli

Kingsnake

Pepper

Backpack

Engine

Present

Teapot

Mount

75-view

50-view

25-view

25-view

50-view

75-view

25-view

50-view

75-view

39.45

38.30

34.32

40.08

38.90

34.63

Ground Truth

Ours

NAF

ASD-POCS

SART

FDK

Mount

22.53

33.09

36.47

36.82

36.80

150-degree

Pancreas

25.

31.14

31.02

34.52

33.27

120-degree

Broccoli

17.28

22.31

21.78

26.38

25.24

90-degree

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we clearly highlight our contribution and scope as a discovery of integration bias problem in 3DGS and a novel 3DGS-based framework for tomographic reconstruction. We discuss details of our claim in Sec. 4 and validate it in Sec. 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitation in Sec. 6. We further show more quantitative and qualitative analysis of limitations in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We carefully derive the X-ray rasterization functions with clear assumptions and numbered formulas as shown in Sec. 4.2.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our work is based on the 3DGS framework with crucial modifications clearly described in Sec. 4. We provide implementation details, including all hyperparameters and training strategies in Sec. 5.1. We also attach our code in the supplementary material for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release the data and code upon acceptance. We attach the code in the supplementary material to reproduce experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all training and testing details, including dataset setting, hyperparameters, and type of optimizer in Sec. 5.1. We discuss hyperparameter analysis in Sec. 5.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While our work and baseline methods involve some random operations, their influence on final results is very limited, so we do not include the error bars. Besides, baseline methods do not provide error bars in their papers for reference.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We clearly provide details of computer resources for our method and baseline methods in Sec. 5.1 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research strictly follows the NeurIPS Code of Ethics. We ensure anonymity for the submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We briefly introduce the potential societal impacts in Sec. 6 and discuss them in detail in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credit the original codes and datasets in our paper and code. The license and terms of use are mentioned along with the code in the supplementary material.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a detailed readme file along with the code in the supplementary material, which shows details about training, license, etc.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces R2-Gaussian, a framework for tomographic reconstruction using 3D Gaussian splatting (3DGS). This framework aims to address the limitations of traditional 3DGS in volumetric reconstruction, specifically for tasks like X-ray computed tomography (CT).

STRENGTHS:
- Identifying and addressing the integration bias in standard 3DGS formulation for volumetric reconstruction
- The proposed R2-Gaussian framework is developed with tailored Gaussian kernels, rectified projection techniques, and a CUDA-based differentiable voxelizer.
- The paper provides simulated X-ray validation, comparing the proposed method against state-of-the-art techniques

WEAKNESSES:
- The proposed method shows results in experimental settings. However, its performance in real-world clinical or industrial scenarios is not thoroughly examined. For instance, real X-ray images are given to reconstruct the CT scan. 

- I am not sure if 75, 50, and 25 views of X-rays are considered sparse-view reconstruction and if it is practical to have 75, 50, or 25 views of X-rays for CT reconstruction. 

- In the proposed method, the kernel formulation removes view-dependent color. However, it cannot model the scattering effect in X-ray.

- In Fig. 7, it is unclear how to get X-3DGS slices and what "queried from three views" means in detail. Why not implement the voxelization on X-3DGS (or vanilla 3DGS) for a fair comparison?

QUESTIONS:
- After the voxelization from Gaussians, can the final density volume be compatible with traditional CT scans and be viewed in CT software?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

comments: SUMMARY:
The paper aims to achieve high tomographic reconstruction performance with a limited number of views in a time-efficient manner.
To this end, the paper modifies 3DGS for X-ray projection by adjusting the rendering equation, correcting 2D projection errors, and using voxelizers for regularization.
The experiments demonstrate the efficacy of the proposed method.

STRENGTHS:
First of all, the paper is well written.
The problem in the current tomographic representation is clearly stated, and the authors' objectives are sufficiently addressed.
The overall structure is easy to understand, and the ablation study covers most of the arising questions.

WEAKNESSES:
One critical weakness of this paper is the existence of prior work using 3DGS for X-ray projection.
Although the paper seems structurally well-written, the paper seems less novel due to the presence of X-Gaussian.
X-Gaussian, which has been accepted to ECCV 2024, employs a similar method.
More importantly, X-Gaussian achieved a PSNR of 43 in Human Organ reconstructions, whereas this paper (R2 Gaussian) achieved a PSNR of 36, despite slightly different experimental settings.
If the authors provide persuasive explanation on the novelty of this paper, I am willing to raise my score.

Additionally, it would be helpful to augment the related works and baseline models.
For example, C^2RV (CVPR 2024) is another recent tomographic representation model.

QUESTIONS:
In the SAX-NeRF paper, the performance gap between SAX-NeRF and NAF is significant.
However, in the R^2 Gaussian paper, the performance gap between them is not significant.
Could you elaborate on why this happens?

LIMITATIONS:
The paper clearly states the limitations and the potential societal impact of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Motivation\n- The authors propose to adapt 3D Gaussian Splatting (3DGS) to sparse-view tomographic reconstruction, i.e., to recover a radiodensity 3D volume from a small set of X-ray images and corresponding sensor information. This is relevant for various clinical and industrial applications.\n\n### Contributions\n- Their _R2-Gaussian_ model iterates over existing 3DGS solutions tuned for XR/CT imaging, proposing a 3DGS initialization scheme better suited for tomographic reconstruction.\n- The authors also correct an integration bias in 3DGS (meant for faster image inference but causing ambiguities in volumetric reconstruction) and provide the corresponding CUDA patch.\n- The proposed _R2-Gaussian_ includes other adaptations (custom densification parameters, voxel-based regularization, simplified isotropic kernels), resulting in an end-to-end CT reconstruction system.\n\n### Results\n- The authors provide extensive qualitative evaluation and quantitatively compare to other NeRF-based and traditional CT reconstruction methods, showing that their method provides a better trade-off between volume accuracy and reconstruction time.\n\n### Relevance\n- Effort in applying 3DGS to XR/CT imaging has grown the past year [6, 27, 39], as 3DGS appears to be a well-suited representation for such applications (due to its compactness, fast convergence, etc.). This work is a meaningful iteration and could benefit the community.\n\nSTRENGTHS:\n_(somewhat ordered from most to least important)_\n\n### S1. Convincing Comparative Evaluation and Qualitative Results\n\n- The quantitative comparison to state-of-the-art CT reconstruction methods appear convincing, with the proposed solution demonstrating a better trade-off between volume accuracy and reconstruction time.\n- The authors provide a lot of meaningful qualitative results, to illustrate theoretical contributions, to highlight the benefits of their method, but also to showcase its limitations. This makes reading this paper the more interesting.\n- An ablation study w.r.t. some of the key contributions and w.r.t. some hyperparameters is also provided.\n- Though limited in number (15 volumes), the authors evaluate on different categories (animal, vegatal, and synthetic targets).\n\n\n### S2. Iterative yet Meaningful Contributions Towards 3DGS for XR/CT\n\n- The discussion w.r.t. the integration bias in vanilla 3DGS is interesting, and the technical solution brought by the authors appear valuable to the community. As mentioned in the paper, their corrected CUDA implementation could benefit other 3DGS works targeting volumetric reconstruction.\n- The authors propose an initialization scheme dedicated to volumetric tomography, as usual 3DGS initialization techniques (e.g., SfM) are not applicable here. This is a relevant contribution, properly described and evaluated (qualitative + quantitative evaluation).\n- The proposed system, tackling xrays-to-CT reconstruction in an end-to-end differentiable manner, is indeed novel. Existing 3DGS models [6, 27, 39] for XR/CT imaging rather focus on digitally-reconstructed-radiograph (DRR) novel-view synthesis (NVS) rather than CT reconstruction.\n\n\n### S3. Sound Theory and Reproducibility\n\n- Background theory is well described by the authors, and the scientific/technical insight of the authors w.r.t. the identified integration bias could benefit the community.\n- The authors provide their model implementation, which appears sound and well-structured (note that I did not try to run the code, but had a look at key files).\n\n\n### S4. Detailed Discussion of Limitations and Contributions\n\n- The authors put significant effort in discussing and illustrating some of their method's limitations (needle-like artifacts inherent to 3DGS, varying convergence time, limited extrapolation ability, etc.), as well as summarizing its impact (possible clinical/industrial applications, benefit of their CUDA code to the CV community, etc.) in Appendices G and H.\n\n\n### S5. Well-Written and Illustrated Paper\n\n- Overall, the paper is nicely structured, written, and illustrated. E.g., Figures 2-4 are helpful to understand the methodology at a glance.\n\nWEAKNESSES:\n_(somewhat ordered from most to least important)_\n\n\n### W1. Lack of Consideration for Real-World Noise and Anisotropic Effects\n\n- The authors claim that \"X-ray attenuation depends only on isotropic density\" [L140] to justify their radiodensity-based model, but this is not entirely correct. While most models generating digitally reconstructed radiographs (DRRs) from CT volumes indeed consider x-ray attenuation as an isotropic phenomenon, this is a approximation. Some x-ray transport effects, such as Compton scattering, are actually anisotropic (but because CT volumes do not inherently contain the material information necessary to accurately simulate Compton scattering, DRR models ignore the anisotropy part) [a]. \nHowever, the authors do claim that they trained their model on DRRs generated using TIGRE [5] configured to simulate Compton scattering [L217-224]. Does it mean that the authors preprocessed the CT volumes to replace the attenuation values by material information (e.g., mapping HU values to a set of predefined materials)? More information on the data generation process would be helpful here.\nIf indeed the model was trained on DRRs containing anisotropic residual noise (c.f. Compton effect's residual impact on XR imaging), but the proposed algorithm itself only consider isotropic attenuation, how does it impact the results? E.g., it could be interesting to generate 2 sets of input DRRs (one generated with the approximated/simplified attenuation model and one more realistic) and compare the final accuracy of the reconstructed CT volumes.\n- The fact that the method is only applied to synthetic inputs (DRRs generated by TIGRE) rather than real, usually noisier, X-ray images is also problematic. The paper would benefit from a real-world evaluation, or at least a discussion on why it was not performed.\n\n\n### W2. Lack of References/Comparisons to SOTA on XR-3DGS\n\n- The authors mention some of the existing 3DGS solutions applied to CT/XR imaging (e.g., X-Gaussian [6], GaSpCT [39], Li et al.'s model [27]) [L89-94] but do not perform any form of comparison with those. The lack of qualitative/quantitative comparison is fair (the authors argue that these models \"cannot generate CT models\" [L91-92], which is somewhat true ; though a comparison on the XR-NVS task could have made this paper stronger). But I would argue that the authors should have better referenced some of these works in the Methodology. E.g., even if less formalized, radiative Gaussians are already presented in X-Gaussian [6] ; and even though it is performed in the pixel domain rather than voxel one, GaSpCT [39] already proposes a total variation loss to regularize their XR-3DGS.\nWhile I still believe that the proposed work is a valuable iteration over these works (by better formalizing and adapting the Gaussian properties and rasterization to CT data), I think the authors should be more transparent w.r.t. the SOTA.\n- References w.r.t. total variation (TV) theory are also missing, making it hard to contextualize the scope of the authors' contribution w.r.t. the objective function.\n\n\n### W3. Somewhat Overstated Contributions\n\n- The CUDA implementation of the radiodensity voxelization seems to be more a technical feat rather than a scientific contribution. While possibly valuable to the community, I do not see any novelty in this module (maybe not as GPU-optimized, but differentiable point-cloud-to-voxel-grid tools already exist, e.g., in PyToch3D).\n- Changes to the adaptive control are unclear/minor, according to the authors' descriptions [L210-214] (i.e., changing the size threshold w.r.t. pruning large Gaussians, editing the density of cloned/split Gaussians).\n- The positive impact of the TV regularization does not appear that statistically significant (+0.32dB for PSNR, +0.009 for SSIM, +1m33s for convergence). Maybe some qualitative results could help grasp its contribution?\n\n\n### W4. Limited Dataset Size\n- The quantitative evaluation is performed on only 15 samples, even though varied. Larger CT datasets are available, e.g. CTPelvic1K [b]. \n\n### W5. Limited Impact of Integration Bias Correction (?)\n- The authors claim that \"this integration bias, though having a negligible impact on imaging rendering, leads to significant inconsistency in density retrieval\" [L184-186], but they only provide qualitative imaging results (Fig. 6) to justify their un-biasing contribution. Additional results could better contextualize the corresponding claims.\n\n\n----\n\n### Additional Reference:\n\n[a] Gao, Zhongpai, et al. \"DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering.\" arXiv preprint arXiv:2406.02518 (2024).\n\n[b] Liu, Pengbo, et al. \"Deep learning to segment pelvic bones: large-scale CT datasets and baseline models.\" International Journal of Computer Assisted Radiology and Surgery 16 (2021): 749-756.\n\nQUESTIONS:\n_see **Weaknesses** for key questions/suggestions._\n\n### Q1. Typo?\n- [L223] Do the authors mean \"Compton scatter\" rather than \"ponton scatter\"?\n\nLIMITATIONS:\nSome limitations and societal impacts are discussed in detail (see **S4** above).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a 3D reconstruction method for sparse-view computed tomography using 3D Gaussian Splatting. The core contribution is the reformulation of the volumetric rendering equation to include view-independent central density estimation. Additionally, the paper introduces a differentiable voxelizer that converts a set of 3D Gaussians into a voxel grid of densities, proving effective in computed tomography tasks.\n\nSTRENGTHS:\n- The paper reveals the view-dependent integration bias in 3D Gaussian Splatting (3DGS), which, to my knowledge, has not previously been reported in the community. This discovery may have a high impact to computer vision. \n- While this is not the first paper to apply 3DGS to computed tomography, it is the first to accurately reconstruct 3D volumes with an image formation tailored specifically for this task. \n- The proposed method is well evaluated and compared against other baseline methods. \n- Exposition is clear. The paper really reads well.\n\nWEAKNESSES:\nI don’t see any particular weakness of the paper.\n\nQUESTIONS:\nI have a couple of questions.\n\nThe paper focuses on sparse-view computed tomography. I am curious about how the method would perform in dense-view scenarios. In industrial CT scanning, capturing a few thousand projections for high-quality microscale geometric details is not uncommon. With a sufficient number of projections, the FDK algorithm typically performs well. How does the proposed method compare to FDK when a large number of projections are used? At what point might it start to underperform, if at all? Would it still outperform in dense-view scenarios?\n\nAdditionally, what are the implications of correcting the integration bias in image-based 3D reconstruction tasks? Would this correction lead to improved 3D reconstructions as well? If not, why?\n\nLIMITATIONS:\nThe only limitation I can think of is the scope of this work; computed tomography represents a relatively niche area in the fields of machine learning and computer vision.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}