Unsupervised Polychromatic Neural Representation for CT Metal Artifact Reduction
Authors: Qing Wu
Lixuan Chen
Ce Wang
Hongjiang Wei
S Kevin Zhou
Jingyi Yu
Yuyao Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Medical Image
Computed Tomography
Metal Arftiacts
Implicit Neural Representation
Unsupervised Learning
TL;DR: We propose Polyner, an unsupervised polychromatic neural representation, that can recover high-quality CT images from metal-corrupted measurements without the need for any extra data.

Abstract: Emerging neural reconstruction techniques based on tomography (e.g., NeRF, NeAT, and NeRP) have started showing unique capabilities in medical imaging. In this work, we present a novel Polychromatic neural representation (Polyner) to tackle the challenging problem of CT imaging when metallic implants exist within the human body. CT metal artifacts arise from the drastic variation of metal's attenuation coefficients at various energy levels of the X-ray spectrum, leading to a nonlinear metal effect in CT measurements. Recovering CT images from metal-affected measurements hence poses a complicated nonlinear inverse problem where empirical models adopted in previous metal artifact reduction (MAR) approaches lead to signal loss and strongly aliased reconstructions. Polyner instead models the MAR problem from a nonlinear inverse problem perspective. Specifically, we first derive a polychromatic forward model to accurately simulate the nonlinear CT acquisition process. Then, we incorporate our forward model into the implicit neural representation to accomplish reconstruction. Lastly, we adopt a regularizer to preserve the physical properties of the CT images across different energy levels while effectively constraining the solution space. Our Polyner is an unsupervised method and does not require any external training data. Experimenting with multiple datasets shows that our Polyner achieves comparable or better performance than supervised methods on in-domain datasets while demonstrating significant performance improvements on out-of-domain datasets. To the best of our knowledge, our Polyner is the first unsupervised MAR method that outperforms its supervised counterparts. The code for this work is available at: https://github.com/iwuqing/Polyner.

Unsupervised Polychromatic Neural Representation for CT Metal Artifact Reduction
Qing Wu Lixuan Chen Ce Wang
Hongjiang Wei
Kevin Zhou
Jingyi Yu Yuyao Zhang
ShanghaiTech University
Shanghai Jiao Tong University
University of Science and Technology of China
Institute of Computing Technology, Chinese Academy of Sciences {wuqing, chenlx1, yujingyi, wangce@ict.aad ct..c.ccac.... cn hongjiang, wei@sji tu. edu. cn
edu, en
skevinzhou@ustc.ede edu. cn
Abstract
Emerging neural reconstruction techniques based on tomography (e.g., NeRF, NeAT, and NeRP) have started showing unique capabilities in medical imaging. this work, we present novel Polychromatic neural representation (Polyner) to tackle the challenging problem of CT imaging when metallic implants exist within the human body. CT metal artifacts arise from the drastic variation of metal's 's attenuation coefficients at various energy levels of the X-ray spectrum, leading to nonlinear metal effect in CT measurements. Recovering CT images from metal-affected measurements hence poses complicated nonlinear inverse prob- lem where empirical models adopted in previous metal artifact reduction (MAR) approaches lead to signal loss and strongly aliased reconstructions. Polyner instead models the MAR problem from nonlinear inverse problem perspective. Specif- ically, we first derive a polychromatic forward model to accurately simulate the nonlinear CT acquisition process. Then, incorporate the implicit neural representation to accomplish reconstruction. Lastly, we adopt a regularizer preserve the physical properties of the CT images across different energy levels while effectively constraining the solution space. Our Polyner is unsupervised method and does not require any external training data. Exper- imenting with multiple datasets shows that our Polyner achieves comparable or better performance than supervised methods in-domain datasets while demon- strating significant performance improvements on out-of-domain datasets. To the best of our knowledge, our Polyner is the first unsupervised MAR method that outperforms its supervised counterparts. The code for this work is available at: wtt.ccoom//i./ee... wwwmiiiiing/oolyeeeeeeeeee
forward model into
Introduction
Computed tomography (CT) is a widely-used imaging technique for clinical diagnosis. CT measure- ments produced by passing X-rays at different angles through the human body. Reconstruction of CT images from these measurements often approximately modeled as a linear inverse problem [1]. However, the attenuation coefficients of the metal materials dramatically vary with the X-ray spectral energy, which largely differs from those of human body tissues. This results in non-negligible nonlinear metal effect in the CT measurements when the body carries metallic implants [2). Hence,
"Coresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

recovering CT images from metal-affect measurements is a complicated nonlinear inverse problem. However. linear inverse problem algorithms, e.g., filtered back projection (FBP) [3], always lead to severe artifacts in the resulting CT images. These artifacts can greatly hamper the accuracy of clinical diagnosis and treatment planning.
Many efforts have been made for CT metal artifact reduction (MAR) task [4]. Most studies [5, 6, 7, 8, 9. 10, 11, 12] simplify this task as linear inverse problem by designing various schemes to replace metal-affected extreme value signals (i.e., metal traces) in the measurements, which unfortunately results in significant degradation of the CT reconstruction due to the signal loss. Currently, supervised deep learning (DL) approaches [7, 8, 13, 14, 15, 16, 17, 6] are the mainstream MAR solutions. Supervised methods typically tend to learn mappings from cortupcor measurements to artifact- free CT images by training deep neural networks on a large-scale external dataset. However, related works face two main limitations. First, collecting numerous paired metal-corrupted measurements and high-quality CT images for training is resource-intensive and laborious. Second, if the metal shapes and the CT acquisition settings differ from those in the training dataset, performance may significantly decline. These challenges significantly limit the practical application of supervised MAR methods in clinical scenarios.
Implicit neural representation (INR) is a new paradigm to solve inverse problems [18]. It uses multi-layer perception (MLP) to model an underlying signal by mapping coordinates to signal responses. The learning bias of the MLPs towards low-frequency signal components [19, 20] is considered as an implicit regularization term for the inverse problems. By approximately simulating the CT acquisition forward model as a linear integral, INR has achieved significant progress in the linear sparse-view CT reconstruction [21, 22, 23, 24, 25, 26]. However, its potential for tackling the nonlinear inverse problem, such as the MAR task, remains unexplored.
In this work, we aim to address the issue of MAR in CT scans from a nonlinear perspective. This allows us to use the complete measurement signals (i.e., including metal traces) and thus achieve better CT reconstruction performance. Specifically, we model the MAR as polychromatic (i.e., multiple X-ray spectral energy levels) reconstruction task of CT image. This task is a highly ill- posed nonlinear inverse problem. To address this, we propose Polychromatic neural representation (Polyner), novel extension of INR for the nonlinear problems. In our Polyner, we make three key designs. First, we derive forward model that accurately simulates the polychromatic nonlinear CT acquisition process. Second, we define a regularization term that effectively constrains the solution space by preserving the physical properties of the CT images across different spectral energy levels. Third, we incorporate our forward model into the INR, granting its capacity to solve the nonlinear inverse problem. Meanwhile, the continuous prior brought by INR also contributes to a more stable solution. Our Polyner is an unsupervised method and does not require any external training data, which makes it potentially applicable to a wide range of CT imaging scenarios.
We evaluate the performance of our Polyner on three datasets, including two simulation datasets and one real-world collected dataset. The experimental results demonstrate that our Polyner performs comparably to the supervised DL method on in in-domain datasets, while significantly outperforming them on out-of-domain datasets. We also conduct extensive ablation studies for several key designs of our method, verifying their effectiveness. To the best of our knowledge, our Polyner is the first unsupervised MAR method superior to supervised counterparts.
The main contributions of this work can be summarised as follows: (1) We propose Polyner, an unsupervised polychromatic neural representation, that can recover high-quality CT images from metal-corrupted measurements without the need for any extra data. (2) We introduce new model perspective for the CT MAR problem, i.e.. recovering polychromatic CT images to mitigate the non- linear metal effect and thus achieving effective MAR. (3) We technically incorporate a polychromatic CT forward model into the INR, enabling the reconstruction of polychromatic CT images.
2 Preliminaries & Related Work
In this section, we first theoretically present the nonlinear metal effect during the CT acquisition process (Sec. 1). Then, we provide brief overview of the literature on the metal artifacts reduction approaches (Sec. $2.2) and implicit neural representations for CT reconstruction (Sec. $2.3).
2

---

Classical model-based algorithms [10, 11, 12] formulate the MAR as an image inpainting task. The metal traces in the measurements are treated as missing and completed by interpolation. However, these model-based methods often produce severe secondary artifacts since the interpolation algorithms ignore the CT physical geometry constraints. With the great success of deep neural networks in low-level vision tasks [31, 32), numerous supervised DI. methods have been proposed [7, 8, 13, 14, 6, 15, 17, 16]. For example, Lin et al. [8] proposed dual domain network (DuDoNet) to 2.1 Nonlinear Metal Effect in CT Measurements
The physical interaction of X-rays passing through an object is governed by Lambert-Beer's law [27, 28), which can be expressed as follows:
[rr, Ea) = Io(Eu) exp (-), ME, (x)
(I)
where r denotes monochromatic X-ray. Fo its energy level, To Fio) the number of photons emitted by an X-ray source, [(r, Eg) is the number of photons received by an X-ray detector, and Me, (x) is the linear attenuation coefficient (LAC) of the observed object at the position x. The LACs serve measure of the object's s ability to absorb the X-rays, with higher values indicating greater absorption ability.
In practice, X-rays often are polychromatic due to limitations in the X-ray source [2, 29]. Consider a polychromatic X-ray composed of multiple monochromatic X-rays belonging an energy level range& E, by combining Eq. (1), the CT measurement data p(r) thus can be written as below:
p(r) = ---h In J8 E)dE Je lo(E)E
= [ [((E) exp [ dE, with (TE) [g (o(')dE
To(F)
where (EE is the normalized energy spectrum that characterizes the distribution of the number of photons To (E) emitted by the X-ray source over the energy level E.
CT reconstruction refers to solve the LACs μ from the measurement data p. is widely recognized that the LACs of human body tissues exhibit slow decreases with increasing the X-ray's energy level, while those of metals undergo sharp variations as a function of energy level [2, 7, 8, 30). Therefore, usual assumption for the human body is that |,,, (x) ja,, (x) R 0, VE,, Es E &, i.e., the LACs of the human body are considered as energy seergy-indeeendent.
Based on this assumption, we can represent the underlying LACs of a human body with metallic implants follows:
(x) + - -M(x)] μ(x), #E (x) _M(x)
where (x) and p(x) denote the LACs of the metal and human body, respectively. M is metal binary mask (M = 1 for the metal region and M = 0 otherwise).
By combining Eq. (2), its CT measurement data be derived
p(r)= = -In In (EE) exp (- (-/. M((xx) vx((x) + - -M(x)] =(x)dx dE
-n (EE)
exp J. Mix) (x))x dE [ 11-
M(x)] [-x
p(x)dx.
(4)
Nonlinear Metal Effect
Linear Integral
This implies that the CT measurement of a human body with metallic implants can be divided into two parts: (1) a linear integral transformation for the human body and (2) a nonlinear transformation for the metallic implants. The latter is also known as the nonlinear metal effect in CT measurements. Therefore, using standard reconstruction techniques, such as FBP [3], which are designed for linear inverse problems, will result in severe metal artifacts in the reconstructed CT image.
2.2
CT Metal Artifact Reduction Approaches

---

((r)
Pv
Σ ((B)).
1
P1
p(r)
Σ
IR?
v(x) E RN
Polychromatic CT Forward Model
Figure I: Overview of the proposed Polyner model. Firstly, we sample coordinates x along an X-ray r. Then, we feed these coordinates into an MLP to predict the corresponding LACs v(x) = (x)(x),.. ... , [x){ at VV energy levels. Thirdly, we leverage differentiable polychromatic CT forward model to transform the predicted LACs into CT measurements p(r). Finally, we optimize the MLP by minimizing the loss between the predicted and real CT measurements.
simuitaneously complete the measurements and enhance the CT images. Wang er al. [13] presented an adaptive convolutional dictionary network (ACDNet), incorporating prior knowledge of metal artifacts into a deep network. These supervised methods achieve state-of-the-ar (SOTA) MAR performance due to their well-designed network architectures and data-driven priors. However, they require large-scale training dataset for supervised learning and often suffer from out-of-domain (OOD) problems [9, 30], severely limiting their usefulness in real-world scenarios. In contrast, our Polyner fully unsupervised method that does not require any external training data. Moreover, both model-based and supervised DL. methods often treat metal traces as missing data, which severely hinders the restoration of image details near metals. Our method decomposes the CT measurement with polychromatic LACs at different energy levels as in Eq. (2), and directly utilizes the complete measurement as input data, which avoids signal loss of the metal traces and thus achieves better MAR performance.
2.3 Implicit Neural Representation for CT Reconstruction
Implicit neural representation (INR) a novel paradigm to continuously parameterize a variety of signals. INR represents an underlying signal continuous function that maps spatial positions to signal responses and uses an MLP network to approximate the complex function. Due to the learning bias of the MLP networks to swwffreqeency signals [19, 20], INR can be used for various vision inverse problems, e.g., view synthesis [33, 34], and surface reconstruction [35, 36]. Recently, many INR-based approaches [22, 23, 24, 25, 37, 21, 38, 39, 40, 26, 41] have been emerged for CT reconstruction. There are two key designs in these methods: (1) representing the unsolved object as a function that maps spatial coordinates to its corresponding monochromatic LCAs at a single energy level, and (2) combining line integral transformation with an MLP network to approximate the function. They have shown excellent CT reconstruction performance benefiting from implicit priors by INR. However, these methods fail to handle the nonlinear CT MAR task because they strictly suppose the unsolved LACs are monochromatic, i.e., energy-independent, leading to the nonlinear metal effect being ignored. In contrast, our method leverages polychromatic forward model to accurately describe the formation of the metal effect and thus enables an effective MAR.
3 Proposed Method
In this section, we propose our Polyner model. We first introduce our modeling for the CT MAR problem (Sec. 1).1). Next. we present a polychromatic CT forward model for simulating the CT acquisition process (Sec. $3.2). Moreover, we define the loss function to optimize our Polyner (Sec. $3.3). Finally, we provide the pipeline recover the reconstruction results (Sec. $3.4). An overview of the proposed Polyner model is shown in Fig. 1.
3.1 Problem Formulation
The LACs of metals to X-rays vary significantly with the spectral energy level of the X-ray, resulting in metal-affect extreme signal values in the measurement data (i.e., nonlinear metal effect discussed in

---

Sec. $2.1). Recovering the underlying CT image from such measurement is complicated nonlinear inverse problem. Existing MAR approaches [5, 7, 8, 9, 10, 11, 12] mostly formulated it as linear inverse problem by removing the metal-affected parts in the measurement.
Instead, aim address the MAR problem from a nonlinear perspective. We suppose that the polychromatic X-rays in the CT acquisition can be decomposed into monochromatic X-rays at N discrete energy levels. Then, we propose to reconstruct the polychromatic CT images at each of the N energy levels (i.e., a total number of N LAC-maps at each monochromatic X-ray for the observed object). This reconstruction task is nonlinear and is independent to the nonlinear metal effect. Hence, the MAR task is modeled as reconstruction problem of the polychromatic CT images.
To accomplish the reconstruction, represent the underlying object as continuous function of spatial coordinate, which can expressed as below:
x = (x, E R + vxx) = (x)( -. [x(x), .M - E IR
(5)
where denotes any spatial coordinate and v(x) is the polychromatic CT image value at that position. Here the item (x) is the LAC of the object to the monochromatic X-ray at energy level E.
However, the function f is very complicated and intractable. Hence, we leverage MLP network Fø (here & denotes trainable weights) to approximate it. other words, we learn neural representation of the function f. The artifacts-free CT images can be reconstructed by feeding all the spatial coordinates into the well-optimized MLP network Fs.
3.2 Polychromatic CT Forward Model
To learn the function f. we optimize the MLP network's trainable weights & to map any spatial coordinate x in space into its corresponding polychromatic LACs vxx). Therefore, differentiable forward model is required for transforming the LACs predicted by the MLP network F& into the CT measurements while also allowing the back- packppropagttion of gradients from the measurement domain to the image domain.
We derive polychromatic forward model for X-ray CT systems in Eq. (2). This forward model can accurately describe the complicated nonlinear CT acquisition process. Our Polyner employs its discrete form to transform the MLP-predicted polychromatic LACs v(x) into the measurement data p. Fig. illustrates its pipeline. For the LACs μ4; at the energy level E;, we first generate its projection value (r) along the X-ray using a linear summation operator. We then leverage the normalized energy spectrum E))) to weight the sum of p. (r) and produce the final measurement data p(r). Formally, the polychromatic CT forward model can be expressed as follows:
N
p(r) -n In Σ n (E,) exp (r)) with (.(r)
Σ [x(x) Ax,
(6)
xer
where Ax represents the distance between adjacent sampled coordinates along the X-ray It is set as the physical resolution (i.e.. voxel size) of the reconstructed CT image. Here, the normalized energy spectrum 17 RN is considered as known prior knowledge. In experiments, we leverage SPEKTR toolkit developed by Punnoose et al. [42] to estimate it.
3.3 Loss Function
Data Consistency Loss. We first compute data consistency (DC) loss that measures the distance between the predicted and measured measurements. The DC loss implemented by E norm, which can be written as below:
Loc =
Σ
(Ix(r) - p(r))la.
R
PER
where R denotes a set of the sampling X-rays r at each training iteration.
-eeeneyd Smooth Loss. As discussed in Sec. $2.1, the LACs of the human body tissues to X-rays exhibit smooth changes with increasing the energy level of the X-rays [7, 8, 30.. We thus propose an energy-dependent smooth (EDS) loss to preserve this physical property. Specifically, for the human body, the proposed EDS loss enforces regularization that constrains the changes between the LACs at any adjacent energy levels to be smooth. We implement this regularization using the L
5

---

norm of the gradient along the energy spectrum. It is approximated by the sum of the absolute error between the predicted LACs at any adjacent X-ray energy levels. Mathematically. the EDS loss can be written as follows:
CEDS
ΣE -M(x) [V
[R]
(rr
XER
(8)
Σ 1 M(x)
Σ (x) xx(x))
[
rER
where M is metal binary mask (M for the metal region and .M- = 0 otherwise).
In summary, our total loss function is defined as below:
== Luc + A LEDS
(9)
where À 2 0 is hyper-parameter that controls the contribution of the EDS regularization CEDS-
3.4
Reconstruction of Artifact-free CT Image
Once the optimization is completed, the MLP network F& is able to predict the polychromatic LACS v(x) {ui (x). (x), ... uv ,Nv(x) (x)) N energy levels for any input spatial coordinate x, meaning the polychromatic CT images can be reconstructed. Given a polychromatic X-ray that consists of multiple monochromatic X-rays at N energy levels, its equivalent monochromatic X-ray's Ei= E,. this work, we aim to solve the CT MAR
energy can be approximated by E problem. Thus, we employ the LACs Mg at the energy level E as our final reconstructed image.
Experiments
In this section, we aim to answer two key questions: (1) Can our unsupervised Polyner compete with supervised DL MAR techniques for the in-domain and out-of-domain datasets? (2) What the impact of the key components our Polyner the model performance? To study these questions, we conduct comprehensive experiments on three datasets. We provide additional experiment details and results in the supplementary material.
4.1 Experimental Settings
Datasets. We conduct experiments on four datasets, including two simulation datasets and two real collected datasets. The first dataset is DeepLesion [43], the most commonly used dataset for the CT MAR evaluation. For this dataset, we extract 200 2D slices of 256x256 256 size from the original 3D CT volumes ground truth (GT). Then, we follow the pipeline described in [30, 7, 8] to synthesize metal-corrupted sinograms as input data. The second dataset the XCOM dataset [44], for which employ two cases provided by Zhang et al. [7]. The third is real-world dataset. As shown in Fig. (Left), we insert a metal paper clip into a walnut sample and scan it with a commercial Bruker SKYSCAN 1276 micro-CT scanner. The fourth is also real-worle dataset. We the same micro-CT scanner to scan a mouse tight containing a metal intramedullary needle. Note that all data solely are utilized for testing purposes since method fully unsupervised.
Baselines & Metries. We employ nine representative CT MAR approaches from three categories as baselines: (1) four model-based algorithms (FBP [3], LI [10], NMAR [11], and ART [45]); (2) three supervised CNN-based DL methods (CNN-MAR [7], DICDNet [14], and ACDNet [13]); and (3) two evaluate unsupervised DL method (ADN [30] and Score-MAR [9]). To ensure a fair comparison,
the five DL models using the checkpoints provided by the authors. Among them, CNN-MAR [7] and Score-MAR [9] are respectively trained on the XCOM [44] and LIDC [46] datasets, while the other three methods are trained on the DeepLesion dataset [43]. We use peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) to quantitatively evaluate the MAR performance.
Implementation Details. For our Polyner, we leverage hash encoding [47] in combination with two fully connected (FC) layers of width 128 to implement the MIP network. A ReL.U activation function then applied after the first FC layer. For the hash encoding [47], we configure its
as follow: L = 16, T 299 8, Nmi 2, and b 2. To optimize the network, we randomly

---

Table Quantitative results of compared methods on the DeepLesion [43] and XCOM [44] datasets. The best and second performances are respectively highlighted in bold and underline.
DeepLesion [43]
XCOM [44]
Category
Method
PSNR
SSIM
PSNR
SSIM
FBP [3]
29.98-13.46
0.7470==0.099
23.76:10.008
0.6506=-0.0200
LI [10]
31.85++3.20
0.8483=0.0726
33.57:0044
0.8866=0.0300
Model-based
ART [45]
32.88=33.63
0.8352=0.0701
32,45+3.76
0.8322=000762
NMAR [11]
32.84:15.07
0.8930=
36.63:10.23
0.9473++00068
CNN-MAR [7]
34.22:12.70
0.9240=0.0375
38.52+0..77
0.9591
Supervised
DICDNet [14]
37.55-42.52
0.9689
32.65+-2.11
0.9449=000037
ACDNet [13]
38.19412.54
0.9675 0.0152
33.04++1.65
0.9049+000089
ADN [30]
33.00:3.22
0.9412=0.0246
25.26:10.94
0.8137=0.0107
Unsupervised
Score-MAR [9]
31.66++3.72
0.9174-0.044
1..92++1.15
0.8054+0.0044
Polyner (Our))
37.57++1.93
0.9754 0.0083
38.74+-0.59
0.9508 +0.0001
FBP
LI
NMAR
CNN-MAR
DICDNet
Deeplesion Dataset (#165)
ACDNet
ADN
Score-MAR
Polyner
GT
NMAR
CNN-MAR
DICDNet
XCOM Dataset (R1)
ACDNet
ADN
Score-MAR
Polyner
GT
Figure 2: Qualitative results of the compared methods on two samples (# 1#124 and #2) of the DeepLesion [43] and XCOM[44] datasets. The red regions denote metals.
sample 80 X-rays, i.e., | 80 in Eqs. (7) & (8), at each training iteration. We set the hyper- parameter to 0.2 Eq. (8). We employ the Adam optimizer [48] with default hyper-parameters and set the initial learning rate to le-3, which decays by a factor of 0.5 1000 epochs. The total hyper-parameters number of training epochs 4000. worth noting that all hyper parameters are tuned on 10 samples from the DeepLesion dataset [43], and then held constant across all other samples.
4.2 Main Results
Comparison on Simulation Data. Table presents the quantitative results. On the DeepLesion dataset [43], ACDNet [13], our Polyner, and DICDNet [14] achieve the top three performances and significantly outperform other baseline models. Specifically, our Polyner produces the second-best performance, trailing slightly behind ACDNet [13] by only -0.62 dB in PSNR. However, on the XCOM dataset [44], we observe that DICDNet and ACDNet trained on the Deeplesion dataset [43] suffer from severe performance drops due to the OOD problem. Their performance is even lower than

---

20/5570.38 3629
23 4850.4853
21/3/0
FBP
ACDNet
10.36/0.7455
22,0070.5484
PSNRYSSIM
Score-MAR
Polyner
Reference
Figure 3: (Left) A Bruker SKYSCAN 1276 micro-CT scanner and walnut sample with metal paper clip. (Right) Qualitative results of the compared methods on the walnut sample data. The red regions denote metals.
that of the model-based NMAR [11] algorithm, with -3.98 dB and -3.59 dB in PSNR, respectively. In contrast, our Polyner obtains the best performance, slightly outperforming CNN-MAR trained on the XCOM dataset [44] by +0.22 dB in PSNR. Fig. 2 shows the qualitative results. Notably, three model-based algorithms (FBP [3], LI [10], and NMAR [11]) and two unsupervised methods (ADN [30] and Score-MAR [9]) cannot yield satisfactory results, exhibiting severe artifacts on both datasets. Conversely, while DICDNet [14] and ACDNet for the DeepLesion dataset [43] yield excellent MAR performances, their results the XCOM dataset [44] include severe artifacts. This contrasts with CNN-MAR [7], which achieves superior results for the XCOM dataset [44] but not for the DeepLesion dataset [43]. These visual inspections are consistent with the above quantitative comparisons. Remarkably, our proposed Polyner presents clean and fine-detail CT reconstructions for both datasets, indicating robustness and superiority over existing MAR methods. More visual results can be found in the supplementary material.
Comparison Real Data. Though our Polyner model performs best the two simulation datasets, it is important to study its performance on real collected CT data. To achieve this, we insert a metal paper clip into a walnut sample and then scan with commercial Bruker SKYSCAN 1276 micro-CT, as shown in Fig 3 (Left). We compare five baselines with our Polyner for reconstructing 2D slice of the sample. The remaining three baselines (NMAR [11], CNN-MAR [7]. and ADN [30]) are not compared because they require prior knowledge of water- segmentation, which is invalid for the walnut sample. Fig. 3 (Right) demonstrates the qualitative results. FBP [3] performs the worst, producing severe shadow artifacts. LI [10] exhibits local deformations in the reconstructed images. Score-MAR R [9] almost fails due to the domain shift problem. ACDNet [13] not effective in completely removing these shadow artifacts, it instead produces an offset in image contrast. In comparison, our Polyner recovers the best visual results in both image details and contrast. We also present the quantitative results in Fig. 3. The proposed Polyner achieves the best performance in two metries. However,
worth noting that the quantitative performances of methods are relatively low (the best PSNR of 22 dB by our Polyner) since the reference CT image cannot be considered GT due to the non non-negin -onneeliii non-rigid deformation caused by the metal paper clip insertion. addition, compare our Polyner with FDK algorithm [49] on the real mouse tight sample scanned by 3D cone-beam CT geometry. Related results are provided in the supplementary material.
4.3 Ablation Studies
Influence Polychromatic CT Forward Model. We explore the efficiency of the polychromatic CT forward model in our Polyner. To this end, we replace it with the linear integral transformation
as in [21, 22, 23, 24, 25, 26). Other model configurations are kept the same for a fair comparison. Fig. 4 demonstrates the qualitative results. From the visualization, it is clear that without the forward model, our Polyner almost fails to handle the shadow artifacts caused by the nonlinear metal effect. In comparison, our full model with the forward model produces a clean CT image that is very close
8

---

Table 2: Quantitative results of our Polyner ablating the poly- chromatic CT forward model on the DeepLesion dataset [43].
Module
PSNR
w/o Poly. CT
w/ Poly. CT
GT
w/o Poly. CT 32.95=4.31
Figure 4: Qualitative results of our Polyner ablating the polychro- matic CT forward model on sample (#6) of the DeepLesion dataset [43]. The red regions denote metals.
w/ Poly. CT
36.87=1.56
Table 3: Quantitative results of our Polyner model over the EDS loss LEDS on the DeepLe- sion [43] dataset.
2.0
Parameter
PSNR
À = 0
22.71=4.97
E= 20
F 39
E 70
E 99
E
120
À 0.1
33.7542.30
Figure 5: Qualitative comparison of the CT images at different en- engy levels using our Polyner model without (Top Row) and with (Bottom Row) the EDS loss CEDS on a sample (#4) of the DeepLe- sion dataset [43]. The red regions denote metals.
À = 0.2
3658
À = 0..
36.83+ 1.52
À 0.4
36.3911.79
to the GT image. We present the quantitative results in Table 2. The results indicate that it contributes to essential improvement of +3.92 dB in PSNR.
Influence of Energy-dependee model performance. To this end, we set different weights À E {0, 0.1, 0.2, 0.3, 0.4) for the EDS loss in Eq. (9). We show the quantitative results in Table 3. The model performance initially improves with an increasing contribution of the EDS loss but later slightly degrades if its contribution keeps increasing. The best performance obtained À 0.2. This because the absence of the EDS loss cannot ensure the smooth changes of the LACs of the body over energy levels. However, excessive bias towards can result the forward model degrading as linear integral transformation, where the polychromatic LACS across all energy levels are identical. Fig. 5 shows the qualitative results. We observe that the CT images at different energy levels randomly change when the EDS loss is not used. In contrast, the regularization can effectively ensure the changes in the LACs are smooth. Moreover, the LACs slowly decrease as the X-ray energy level increases, which is generally consistent with the experimental findings reported in previous works [50].
Smooth Loss. We investigate the influence of the EDS loss on the
We investigate how the number of discrete energy levels N affects the model performance. We set the energy range to [20, 120) and uniformly sample the N (50, 76, 86, 101} energy levels from the range. We also resample the normalized energy spectrum 7 E T to match these levels. Table 4 shows the quantitative results, which indicate that increasing the energy resolution improves the MAR performance. We present the qualitative results in Fig. 6, which show that our Polyner with higher energy resolution (N 101) produces better CT results in terms of both image details and contrast. Our method approximates polychromatic X-rays with discrete energy levels, introducing approximation error. Therefore, improving the energy's 's resolution may help to reduce this error and improve the model performance.
Influence of Number Energy Levels.
5 Conclusion & Limitation
We present Polyner, novel method for the nonlinear MAR problem. The proposed Polyner follows the unsupervised learning paradigm and does not require any external training data, which greatly makes it useful in clinical scenarios. Our Polyner learns neural representation of polychromatic

---

Table 4: Quantitative results of our Polyner ablating the discrete energy levels N on the DeepLesion [43] dataset.
# Energy Levels
PSNR
N 50
30.98+1.022
N
101
GT
N = 76
34.99+1.30
N 86
36.15+1.38
Figure 6: Qualitative results of our Polyner ablating the discrete energy levels N on a sample (#9) the Deeplesion dataset [43]. The red regions denote metals.
N = -101
36.87:1..56
CT images fundamentally avoid the nonlinear metal effect causing metal artifacts. It thus can reconstruct clean CT images from metal-affect measurements. Extensive experiments on three CT datasets demonstrate that our Polyner yields comparable or even better supervised DL methods.
Though the proposed Polyner achieves excellent MAR performance, there still are some limitations. Firstly, our Polyner is case- case-specific, meaning independent model has be optimized for each sample. Technically, the optimization of the Polyner for CT image of 256x256 size requires about 2 minutes single NVIDIA RTX TITAN GPU (24 GB). In addition, our current Polyner model is based on 2D fan-beam and 3D cone-beam CT, while more advanced types of beams (e.g., helical CT) not implemented.
6
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grants No. 62071299 and MoE Key Lab of Intelligent Perception and Hauhan-Machine Collaboration (Shang- haiTech University). Ce Wang was supported by the National Natural Science Foundation China under Grants No. 62301532, in part by the Natural Science Foundation of Jiangsu Province under Grant No. BK20230282
References
[1] Ge Wang, Hengyong Yu, and Bruno De Man. An outlook x-ray research and development. Medical physics, (5() 2008
[2] Jin Keun Seo and Eung Je Woo. Nonlinear inverse problems in imaging. John Wiley Sons, 2012.
[3] Avinash Kak and Malcolm Slaney. Principles of computerized tomographic imaging. SIAM, 2001.
[4] Lars Gjesteby, Bruno De Man, Yannan Jin, Harald Paganetti, Joost Verburg, Drosoula Gi- antsoudi, and Ge Wang. Metal artifact reduction in ct: where are we after four decades? leee Access, 4:5826-5849, 2016.
[5] Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-20 1992.
[6] Muhammad Usman Ghani and W Clem Karl. Fast enhanced et metal artifact reduction using data domain deep learning. IEEE Transactions on Computational Imaging, 6:181-193, 2019.
[7] Yanbo Zhang and Hengyong Yu. Convolutional neural network based metal artifact reduction in x-ray computed tomography. JEEE transactions on medical imaging. 3776):130--138) 2018.
[8] Wei-An Lin, Haofu Liao, Cheng Peng, Xiaohang Sun, Jingdan Zhang, Jiebo Luo, Rama Chellappa, and Shaohua Kevin Zhou. Dudonet: Dual domain network for et metal artifact reduction. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 10512-10521, 2019.
10

---

[9] Yang Song,
Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medi- cal imaging with score-based generative models. In International Conference on Learning Representations, 2022.
[10] Willi A Kalender, Robert Hebel, and Johannes Ebersberger. Reduction of ct artifacts caused by metallic implants. Radiology, 164(2):566-777, 1987.
[11] Esther Meyer, Rainer Raupach, Michael Lell, Bernhard Schmidt, and Mare KachelrieB. Normal- ized metal artifact reduction (nmar) in computed tomography. Medical physics, 37(10):5482- 5493, 2010.
[12] Abolfazl Mehranian, Mohammad Reza Ay, Arman Rahmim, and Habib Zaidi. X-ray et metal artifact reduction using wavelet domain [_(0] sparse regularization. JEEE transactions on medical imaging, 3779::707-1122. 2013.
[13] Hong Wang. Yuexiang Li, Deyu Meng. and Yefeng Zheng. Adaptive convolutional dictionary network for ct metal artifact reduction. In The 31 International Joint Conference Artificial Intelligence. IEEE, 2022.
[14] Hong Wang, Yuexiang Li, Nanjun He, Kai Ma, Deyu Meng, and Yefeng Zheng. Dicdnet: Deep interpretable convolutional dictionary network for metal artifact reduction in images. IEEE Transactions on Medical Imaging, 41 (44:8869,-880 2021.
[15] Lars Gjesteby, Qingsong Yang. Yan Xi, Hongming Shan, Bernhard Claus, Yannan Jin, Bruno De Man, and Ge Wang. Deep learning methods for ct image-domain metal artifact reduction. In Developments in X-ray Tomography XI, volume 10391, pages 147-152. SPIE, 2017.
[16] Lequan Yu, Zhicheng Zhang, Xiaomeng Li, and Lei Xing. Deep sinogram completion with image prior for metal artifact reduction in ct images. IEEE transactions on medical imaging, 40(1):228-233) 2020.
[17] Jianing Wang. Yiyuan Zhao, Jack H Noble, and Benoit M Dawant. Conditional generative adversarial networks for metal artifact reduction in et images of the In Medical Image Computing and Computer Assisted Intervention-MICO 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pages 3-11. Springer, 2018.
[18] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. In Computer Graphics Forum, volume 41, pages 641-676. Wiley Online Library, 2022.
[19] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International Conference Machine Learning, pages 5301-5318 PMLR, 2019.
[20] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: arXiv: 1901006300.. 2019.
[21] Darius Rückert, Yuanhao Wang, Rui Li, Ramzi Idoughi, and Wolfgang Heidrich. Neat: Neural 2022.
adaptive tomography. ACM Transactions on Graphics
[22] Liyue Shen, John Pauly, and Lei Xing. Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[23] Albert W Reed, Hyojin Kim, Rushil Anirudh, K Aditya Mohan, Kyle Champley, Jingu Kang, and Suren Jayasuriya. Dynamic ct reconstruction from limited views with implicit neural representations and parametric motion fields. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2258-22666 2021.
[24] Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov. Coil: Coordinate- internal learning for tomographic imaging. IEEE Trunsactions on Com- putational Imaging, 1:1400-1412, 2021.
11

---

[25] Guangming Zang, Ramzi Idoughi, Rui Li, Peter Wonka, and Wolfgang Heidrich. Intratomo: self- supervised learning- based tomography via sinogram synthesis and prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1960-1970, 2021.
[26] Qing Wu, Ruimin Feng, Hongjiang Wei, Jingyi Yu, and Yuyao Zhang. Self-supervised co- ordinate projection network for sparse-viee view computed tomography. IEEE Trunsuctions on Compulational Imaging, 9:517-529, 2023.
[27] Johann Heinrich Lambert. Photometria sive de mensura et gradibus luminis, colorum et umbrae. Klett, 1760.
[28] August Beer. Bestimmung der absorption des rothen lichts in farbigen flussigkeiten. Ann. Physik, 162:78-88 88, 1852.
[29] Emil Y Sidky, Yu Zou, and Xiaochuan Pan. Impact of polychromatic x-ray sources on helical, cone-beam computed tomography and dual-energy methods. Physics in Medicine & Biology, 49(11): 2004.
[30] H. Liao, W. Lin, K. Zhou, and J. Luo. Adn: Artifact disentanglement network for unsupervised metal artifact reduction. IEEE Transactions on Medical Imaging, 2019.
[31] Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(10))336 2020.
[32] Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia-Wen Lin. Deep learning on image denoising: An overview. Neural Networks, 131:251-275, 2020.
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[34] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Barain-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[35] Lior Yariv, Jiatao Gu. Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-498 4815, 2021.
[36] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171
2021.
[37] Ruyi Zha, Yanhao Zhang, and Hongdong Li. Naf: Neural attenuation fields for sparse-view obet reconstruction. In Medical Image Computing and Computer Assisted Intervention-MICCAl 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings. Part VI, pages
Springer, 2022.
[38] Francisca Vasconcelos, Bobby He, Nalini Singh, and Yee whye Teh. Uncertainr: Uncertainty quantification of end-to-end implicit neural representations for computed tomography. arXiv preprint 2081 2022.
[39] Bowen Song, Liyue Shen, and Lei Xing, Piner: Prior-informed implicit neural representation learning for test-time adaptation sparse-view et reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1928-1938, 2023.
[40] Kunal Gupta, Brendan Colvert, Zhennong Chen, and Francisco Contijoch. Difir-ct: Distance field representation to resolve motion artifacts in computed tomography. Medical Physics, 50(3):1349-1333 -1366 2023.
[41] Qing Wu, Xin Li, Hongjiang Wei, Jingyi Yu, and Yuyao Zhang. Joint rigid motion correction and sparse-view et via self-calibrating neural field. 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1-5. IEEE, 2023.
12

---

[42] J Punnoose, J Xu, A Sisniega, W Zbijewski, and JH Siewerdsen. Technical note: spektr 3.0 computational tool for x-ray spectrum. Medical Physics, 43(8):4111-4777. 2016.
[43] Ke Yan, Xiaosong Wang, Le Lu, Ling Zhang, Adam P Harrison, Mohammadhadi Bagheri, and Ronald M Summers. Deep lesion graphs in the wild: relationship learning and organization of sigmificant radiology image findings a diverse large-scale lesion database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9261-9270, 2018.
[44] Martin J Berger. Xcom: Photon cross sections database, version 1.4. http:/pphysies. nist.
gov/xcom, 2009.
[45] Richard Gordon, Robert Bender, and Gabor T Herman. Algebraic reconstruction techniques (art) for three-dimensional electron microscopy and x-ray photography. Journal of theoretical Biology, 29(3):471-488 1970.
[46] Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric Hoffman, al. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on et seans. Medical physics, 38(2):915-931, 2011.
[47] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41)) jul 2022.
[48] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
[49] Lee A Feldkamp, Lloyd C Davis, and James W Kress. Practical cone-beam algorithm. Josa a, 1(6):612-619, 1984.
[50] RC Chen, Renata Longo, Luigi Rigon, Fabrizio Zanconati, Alessandro De Pellegrin, Fulvia Arfelli, Diego Dreossi, RH Menk, Vallazza, TQ Xiao, al. Measurement the linear attenuation coefficients of breast tissues by synchrotron radiation computed tomography. Physies in Medicine & Biology, 55(17):4993, 2010.
13

---

Supplementary Material- Unsupervised Polychromatic Neural Representation for CT Metal Artifact Reduction
Figure 7: Ten shapes of metals for synthesizing metal-corrupted measurements in the DeepLesion
[43] dataset. These metals are supposed as Titanium. The white regions denote metals.
0.03
0.03
0.02
n(E)
n(E)
0.01
0.01
20
40
60
100
120
20
40
100
10
20
30
50
60
E (KeV)
(KeV)
E (KeV)
Figure 8: (Left) The simulated spectrum within an energy range [20, 120] for synthesizing metal- corupped measurements of the DeepLesion [43] and XCOM [44] datasets. The spectrums estimated within the energy ranges of [O, 100] and [O, 60] estimated by the SPEKTR toolkit [42] for the real walnut sample (Middle) and mouse thigh (Right).
A Additional Details of Datasets
In this work, perform experiments four datasets: DeepLesion [43], XCOM [44], and our Walnut Sample. Deepl and XCOM are simulation datasets, while Walnut Sample and Mouse Thigh are real-world datasets. Note that method fully unsupervised, and thus all the data exclusively used for testing purposes.
DeepLesion. To simulate metal-corrupted measurements in the DeepLesion dataset [43], we follow pipelines outlined in [30, 7, 8]. For our experiments, we extract 200 2D images from the DeepLesion dataset test GT samples [43]. As depicted in Fig. 7, we leverage ten shapes of metallic implants from [30, 7, 8] and consider them Titanium. To simulate the X-ray source, we employ polychromatic X-ray with energy range of [20, 120] KeV and a minimum energy unit of KeV. The number of photons emitted by the X-ray source is set to 2x 10². The normalized energy spectrum n of the X-ray source is illustrated in Fig. (Left). We adopt geometry, and the detailed parameters are provided in Table 5. Additionally, we incorporate Poisson noise and consider the partial volume effect in the sinogram domain during the simulation process.
equiangular fan-beam CT acquisition
XCOM. For the XCOM dataset [44], we use two samples provided by Zhang al. [7]. These two cases are simulated using two 2D clean CT images sourced from the XCOM [44] database. Zhang
14

---

Table 5: Detailed parameters of the acquisition geometry for the four datasets.
Simulation Datasets
Real Datasets
Parameters
DeepLesion [43]
XCOM [44]
Walnut
Mouse Thigh
Type of geumetry
2D fan-beam
2D fan-beam
2D fan-beam
3D cone-beam
Source Voltage (kV)
120
120
100
60
Source Current (uA)
200
200
Exposure Time (ms)
276
730
Image size
256x256
512x512
650 650
x00 150
Voxel (mm)
1x1
0.8
x.05
0.06x0.06x0.06
Angle range (*)
[0. 360)
(603
(0.360)
360)
The number the angles
360
984
720
900
Angular spacing (?)
0.1
0.057
Detector spacing (mm)
0.069
0.069
Distance from source center (mm)
362
743
92.602
92.602
Distance from center to detector (mm)
362
743
65.946
65.946
al. [7] consider polychromatic X-ray source with an energy range of [20, 120] KeV and minimum energy unit of KeV. The corresponding normalized energy spectrum 7j depicted Fig. (Leff). In oder to generate metal-corrupied sinograms, an equiangular fan-beam CT acquisition geometry employed, and the geometry parameters are specified Table 5. Similar to the DeepLesion [43] dataset, Zhang et al. [7] also simulate Poisson noise and consider the partial volume effect in the sinogram domain during the simulation process.
Walnut Sample. To assess the performance of our proposed method on real CT data, we employ commercial Bruker SKYSCAN 1276 micro-CT scanner to sean a walnut sample that contains a metal paper clip. Detailed parameters of the acquisition geometry can be found in Table 5. To estimate the X-ray spectrum of the micro-CT scanner, we leverage the SPEKTR toolkit [42]. The estimated spectrum is illustrated in Fig. 8 (Middle).
Mouse Thigh. We also scan a mouse thigh containing metal intramedullary needle on the same micro-CT scanner. This sample is 3D cone-beam data. Detailed parameters of the acquisition geometry are shown in Table 5. We leverage the SPEKTR toolkit [42] to estimate the X-ray spectrum of the micro-CT scanner. The estimated spectrum is illustrated in Fig. 8 (Right).
B Additional Details of Baselines
In experiments, compare proposed method against eight baseline MAR approaches. Notably, for the five DL-based methods, we evaluate their performance using the pre-trained models provided by the respective authors.
FBP. The FBP [3] conventional approach used for reconstructing CT images. involves re-projecting the acquired sinogram data onto the image domain using the corresponding projection angles and geometric parameters to obtain an approximate estimate of the unknown image. In our experiments, we use the in-build function ifanbem in MATLAB (https://wal mathworks cn/ help/ http/immages/reff iifabbeam. html hwtthtteeuuuuueeee (eamll/rrggstteedmmaaaarc.a
A simple strategy to mitigate metal artifacts in CT imaging involves the direct linear interpola- tion of the sinogram [10] to fill in the regions affected by metal. This approach does not require any network training but may result in imperfect sinogram completion, which in turn could introduce un- desired artifacts the reconstructed image. In our experiments, we use the implementation provided by Zhang et al. [7] (https: :// //ti//gitt /gggttu...... comm/ynboozango07/O0oo MAin-MAR//bA MAR/blob/ http://bbbb/mmteeeeeee master/cnnmar)
LI.
15

---

The NMAR [11] introduces a generalized normalization technique that extends previously developed interpolation
NMAR.
MAR techniques (e.g.. LI [10]), which also does not require any network training. Specifically, it normalizes the projections before interpolation based on forward projections of a prior image obtained through multi-threshold segmentation of the initial image. In our experiments, we use the implementation provided by Zhang et al. [7] (https: : //gi httptttb..././///g///////g thub. com/ yanbozhang007/Caa CNN MAR ht/t//rre//m/trr/e///e// /cnemar))...
CNN-MAR. Zhang el al. [7] proposed a CNN-based MAR framework, which uses CNN to estimate a prior image and subsequently apply sinogram correction. However, despite the strong representation ability of CNNs, these approaches still susceptible to secondary artifacts resulting from inconsistent sinograms. In our experiments, we use its official implementation and pre-trained model (https://gg) //ggth//g./tt////// com/yanbozhang007/C0e MAR// Www.MMARRRttteeeM- cree/master/ coneear)..
DICDNet. Wang et al. [14] propose a deep interpretable convolutional dictionary network (DICD- Net) for the MAR task, which explicitly formulates the prior structures underlying metal artifacts in CT images as a convolutional dictionary model. our experiments, we use its official implementation and pre-trained model (https: //gi httgggttt......./
wtt.o//wangg//DIDDeee////// tett///rre/mmmmm/ee /main).
ACDNet. Similarly to DICDNet [14], the adaptive convolutional dictionary network (ACDNet) [13] explicitly encodes the prior observations underlying the MAR task into adaptive convolutional dictionary network. In our experiments, we use its official implementation and pre-trained model tattps://githur com womm/hogwwnngg//AAe //CDNNet).
ADN. The ADN [30] is an unsupervised learning approach for the MAR problem. Specifically, takes unpaired metal-corrupted and clean CT images inputs learn the transformation between these two distributions. In our experiments, we use its official implementation and pre-trained model tattt//ree/mmstttt
(https://githen com/
Score-MAR.
Song et al. [9] demonstrated that unconditional diffusion models can be adapted to various inverse problems, such the MAR. Specifically, it learns the prior distribution of metal- free CT images with generative model in order to infer the lost sinogram in the metal-affected regions. In our experiments, we its official implementation and pre-trained model (https: lgggitbb.ccm/yygg-goortott wow.yaannnno//coreaaaaaaeeeeeeee http:/ore__ierreeerrollleee
C Additional Visual Results
Figs. 9, 10, 11, 12, and demonstrate some additional visual results. We observe that the proposed Polyner generally obtains the best MAR results.
D Broader Impacts
Our Polyner is expected to have significant broader impacts in the field of medical imaging. By effectively reducing metal artifacts in CT scans, research has the potential to improve diagnostic accuracy. leading to more precise diagnoses and enhanced patient care. However, it important to address potential limitations and concerns, such as the possibility of introducing false positive or negative results. Thorough evaluation and validation of our method crucial to ensure its reliability and minimize any adverse effects. Overall, work contributes advancing medical imaging technology and has the potential to positively impact healthcare.
16

---

FDK
Polyner (Ours)
2D Projection
Figure 9: (Left) A sample among 2D projections mouse thigh containing metal intramedullary needle scanned by the micro-CT scanner. (Right) Qualitative results of FDK [49] and our Polyner on the sample. Note that the acquisition geometry is the 3D cone beam. The reconstructed images have size of x00 150. Our Polyner takes about 32 minutes single NVIDIA RTX TITAN GPU, The red regions denote the metal needle tubing. This data collection is approved ethically.
FBP
LI
NMAR
CNN-MAR
DICDNet
ACDNet
ADN
Score-Mo MAR
Polyner (Ours)
GT
Figure 10:
Qualitative results of the compared methods on a sample (#74) of DeepLesion [43] dataset. The white regions denote metals.
17

---

FBP
NMAR
CNN-MAR
DICDNet
ACDNet
ADN
Score-MAR
Polyner (Ours)
GT
Figure 11: Qualitative results of the compared methods on a sample (# 162) of DeepLesion [43] dataset. The white regions denote metals.
18

---

NINAMAR
DICONER
FBP
L
NMAAR
ADN ACONET
LODC-MMAA
Polyner (Ours)
GT
Figure 12: Qualitative results of the companres methods on a sample (# 158) of DeepLesion [43] dataset. The white regions denote metals.
19

---

FBP
NMAR
CNN-MAR
DICDNet
ACDNet
ADN
Score-MAR
Polyner (Ours)
GT
Qualitative results of the compared methods on a sample (#1) of XCOM [44] dataset. The white regions denote metals.
Figure 13:
20


Reviews:
[{"review_id": "HV5PciL1AW", "reviewer": "Reviewer_ibgY", "summary": "The paper proposes a new implicit neural representation (INR)-based polychromatic x-ray CT reconstruction technique called Polyner. In normal CT reconstruction, the tissues of the body do not vary substantially in their attenuation coefficients, so the overall forward operator can be simply modeled as linear. This changes in the presence of metal implants, because the attenuation of metal implants depends on the energy of the input X-rays, giving a polychromatic attenuation response. Several methods have been proposed for polychromatic CT reconstruction, but these methods often manifest as inpainting methods that suffer from out-of-distribution generalization issues.\n\nThe present paper addresses this issue by using a NeRF-like model for estimating the attenuation coefficients in the body. The model is optimized by passing the predicted attenuations through a forward integral model and minimizing the distortion vs. the measurements. The forward model is adapted for metal vs. non-metal regions via mask.\n\nResults presented in the paper are promising: the proposed Polyner method is competitive with past methods on in-domain test datasets in terms of PSNR and SSIM metrics on the DeepLesion and XCOM datasets. Competing methods have wide variances in performance among the two datasets, whereas Polyner is more stable. On a new CT dataset consisting of a walnut scan with a metal paperclip, where Polyner further establishes is strong robustness. In ablations, the paper establishes the importance of decisions taken with respect to the forward model and loss functions.", "questions": "1. Most ablations consider loss functions and forward models - did the authors consider modifications to the INR architecture, or is this not an area with large expected benefits?\n2. Please specify more details on the binary mask - it was not clear to me how this is calculated or used from the text.", "limitations": "Limitations are listed with the discussion. I normally prefer a separate listing, but I do not have a major objection to this presentation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality**\n- The paper presents a novel approach for the metal artifact reduction problem. Although implicit neural representations have been used for previous CT reconstruction tasks, the metal artifact reduction problem is substantially different in terms of the physics under consideration and the artifacts that can manifest.\n\n**Quality**\n- The paper includes all the expected experiments, examining multiple simulation datasets as well as new real-world data, providing strong experimental validation of the proposed method.\n- Comparison methods seem up to date, including methods as recent as 2022.\n\n**Clarity**\n- The paper is clear in its motivations for the metal artifact reduction problem and the justification for the INR approach.\n\n**Significance**\n- Providing a robust reconstruction method for CT with metal artifacts is a long-standing goal of the CT field, and it seems that this paper makes a god contribution towards this goal.", "weaknesses": "**Originality**\n- The presence of previous INR-style methods in CT arguably reduces the originality of the current paper, but in my opinion not in a significant manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the valuable comments. We are encouraged by your recognition of our work. Below, we provide point-to-point responses to address your concerns.\n\n---\n\n**Q1. Most ablations consider loss functions and forward models - did the authors consider modifications to the INR architecture, or is this not an area with large expected benefits?**\n\n**A1.** The INR architectures usually consist of an encoding module and an MLP network. The encoding module transforms low-dimensional coordinates into high-dimensional embeddings, significantly enhancing the fitting capacity of the subsequent MLP network. In our Polyner model, we employ the SOTA hash encoding for embedding, which has become a standard module in many INR-based approaches [1]. Here, we compare it with position encoding [2] and Fourier encoding [3] on the DeepLesion dataset. From Table R1, we can see that hash encoding achieves the best performance in terms of both reconstruction quality and speed. Qualitative results are shown in Figure R2 of the PDF file of the Global Response. \n\nWe will include this ablation study for the INR architecture in the supplementary material.\n\nIn summary, the INR architectures do impact the model performance, particularly in terms of optimization speed. Hence, adopting a more powerful architecture can significantly improve the proposed Polyner model.\n\n|\t|Position Encoding|\tFourier Encoding|\tHash Encoding|\n| :---:| :---:| :---:| :---:|\n|PSNR\t|34.55±1.64|\t35.07±1.42|\t**36.87±1.56**|\n|Average Time\t|10 min.|\t8 min.|\t**2 min.**|\n\n*Table R1: Qualitative results of our Polyner implemented by three different INR architectures on the DeepLesion dataset.*\n\n> [1] Tewari, Ayush, et al. \"Advances in neural rendering.\" Computer Graphics Forum. 2022.\n\n> [2] Mildenhall, B., et al. \"Nerf: Representing scenes as neural radiance fields for view synthesis.\" ECCV. 2020.\n\n> [3] Tancik, Matthew, et al. \"Fourier features let networks learn high frequency functions in low dimensional domains.\" NeurIPS. 2020.\n---\n\n**Q2. Please specify more details on the binary mask - it was not clear to me how this is calculated or used from the text.**\n\n**A2.** Due to the substantial distinctions in the linear attenuation coefficients (LACs) between metallic substances and biological tissue, we applied a straightforward threshold segmentation to the CT images reconstructed by FBP to generate these binary metal masks. \n\nIn the EDS regularization term (Eq. 8), we use these masks to distinguish between metal and body regions and thus can enforce a smooth prior along the energy spectrum for the body area. We will include these details in the revised manuscript."}, {"review_id": "tkeiMop9kY", "reviewer": "Reviewer_KPnC", "summary": "The paper with title: Unsupervised Polychromatic Neural Representation for CT Metal Artifact Reduction presents an Implicit neural representation-based method for CT metal artifacts reduction, outperforming existing supervised and unsupervised approaches.\n", "questions": "I have put my questions in the weakness section.", "limitations": "1. The inference time is long if given limited performance improvements. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a novel INR-based method for CT metal artifacts reduction. \n2. The authors present a non-linear forward model to model the metal artifacts, and leverages it as signal domain loss function.\n3. I appreciate the real-scan results which demonstrate the results on prospective corrupted dataset.", "weaknesses": "1. Regarding the results for the real-data - I noticed that for real-data, the gap or improvements of Polyner is not as significant as the simulated dataset, ACDNet is also doing a decent job, could you elaborate on this?\n2. Regarding the forward model, from the ablation studies, including the non-linear components contribute to the improved performance, however, the forward model is still not perfect without considering some system attributes. Can you elaborate on this aspect?\n3. lack of quantitative results on real-data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. We are pleased to receive your positive feedback. Below, we provide point-to-point responses to address your concerns.\n\n---\n\n**Q1. Regarding the results for the real-data - I noticed that for real-data, the gap or improvements of Polyner is not as significant as the simulated dataset, ACDNet is also doing a decent job, could you elaborate on this?**\n\n**A1:** For MAR methods, the performance is significantly influenced by the volume and shape of the metallic implant. In our real-data scanning, the metallic object exhibits a compact structure with a volume much smaller than those in the simulation data. The metal streaking artifacts are narrow and are not as complicated as that in the simulation data, as shown in the reconstruction result by FBP in Figure 3, while ACDNet takes the images by FBP as one of the inputs. Therefore, the reconstruction results of ACDNet are also acceptable. \n\nHowever, residual streaking artifacts are still observed by the results of ACDNet, and the signal intensities obviously deviate from the reference image. In contrast, our Polyner provides high-fidelity reconstruction with neglectable streaking artifacts.\n\n---\n\n**Q2. Regarding the forward model, from the ablation studies, including the non-linear components contribute to the improved performance, however, the forward model is still not perfect without considering some system attributes. Can you elaborate on this aspect?**\n\n**A2:**  CT acquisition is a complicated process with various applications. Polyner is primarily proposed for medical CT, the most widely beneficial application among all application scenarios. In medical CT, commercial scanners adhere to guidelines for biological CT scans, characterized by controlled conditions: normal-dose X-rays, full projection views, exposure time, standard SNR, well-calibrated energy sources, and signal sensors. In such a setup, we propose our forward model primarily focuses on simulating the nonlinear effects encountered when scanning a human body with metal implants. We hold that in a relatively stable environment, the nonlinear forward model covers the most critical contributors in MAR CT reconstruction. The descent performance on real data also proves the efficiency of the forward model.\n\nMoreover, our Polyner model can easily be extended to tackle more complicated CT scanning situations. For instance, we could incorporate explicit regularization terms such as total variation into the loss function (Eq. 9) to compensate for the low SNR data acquisition in low dose CT condition [1]; we could combine a reprojection strategy for sparse view CT condition [2]; we could jointly optimize the INR network and a transformation matrix when the scanner sensor calibration is not perfect to correct motion between each projection [3].\n\n> [1] Zang, Guangming, et al. \"IntraTomo: self-supervised learning-based tomography via sinogram synthesis and prediction.\" CVPR. 2021.\n\n> [2] Wu, Qing, et al. \"Self-supervised coordinate projection network for sparse-view computed tomography.\" IEEE TCI (2023).\n\n> [3] Wang, Zirui, et al. \"NeRF--: Neural radiance fields without known camera parameters.\" arXiv preprint arXiv:2102.07064 (2021).\n\n---\n\n**Q3. Lack of quantitative results on real-data.**\n\n**A3:** In our real-data experiment, we only provide qualitative results. The main reason is that the reference CT image cannot be completely considered a ground truth due to non-rigid deformation caused by the insertion of the metal paper clip. This misalignment will affect the accuracy of quantitative metrics. Here, we calculate the quantitative results in Table R2. Our Polyner still performs the best. We will add these quantitative results to the supplementary material in the revised manuscript.\n\n|\t|FBP|\tLI|\tACDNet|\tDIDCNet|\tPolyner (Ours)|\n| :---: |:---: |:---: |:---: |:---: |:---: |\n|PSNR|\t20.66|\t21.94|\t21.53|\t21.61|\t**22.00**|\n|SSIM|\t0.3623|\t0.4853|\t0.3592|\t0.3562|\t**0.5484**|\n\n*Table R2: Quantitative results of the compared methods on the real collected walnut sample.*\n\n---\n\n**Q4. The inference time is long if given limited performance improvements.**\n\n**A4:** See Q1 in the Global Response, please. And we would like to emphasize our Polyner contributes the most in proposing a fully unsupervised MAR method. At the same time, the performance is superior compared with existing supervised MAR methods.\n\n\n"}, {"review_id": "wMwUjdQpuH", "reviewer": "Reviewer_oo65", "summary": "This paper introduces Polyner, an extension of implicit neural representation to a nonlinear inverse problem with a forward model that simulates the polychromatic nonlinear CT acquisition process. This design allows Polyner to reduce metal artifacts without external training data and exhibits better generalization to out-of-domain (OOD) data. Experimental results showcase improvements of Polyner on OOD datasets and clinical data while being competitive with state-of-the-art supervised methods.", "questions": "1. It is important to clarify whether the DL methods (excluding CNNMAR) were fine-tuned on your dataset to match the geometry settings, following the training procedures described in their papers. If not, please provide the fine-tuned results for evaluation.\n\n2. Please specify whether the PSNR and SSIM metrics were calculated based on specific windowing or attenuation coefficients.\n\n3. Suggest changing the color scheme of the visual results. While it may be suitable for the ablation study to highlight details, grayscale images are more widely accepted in the field of CT imaging.\n\n4. Sinogram inpainting-based methods should be reproduced and compared, which can show the advantages of the proposed polychromatic model. \n\n5. Some statements are not accurate. (a) L36“..remove metal-affected extreme value signals....” Existing methods used different methods to replace these values. (b) L41 “collecting a large number of artifacts-free CT images...”, collecting such a dataset is easy. Instead, collecting a paired dataset is challenging. (c)  In the caption of Fig. 3, swap left and right.\n", "limitations": "In addition to these limitations in the paper, there is another limitation: When coming to 3D reconstruction, the proposed methods may produce discontinuous z-axis since it is a case-specific method for each slice.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The motivation is technically sound, and the proposed unsupervised method is interesting compared to the previous supervised paradigm. Taking into account the important knowledge of CT is reasonable. \n\n2. The performance on OOD dataset and clinical images demonstrates the OOD capacity, though the quantitative evaluation (see weakness) is unfair. The ablation study confirms the effectiveness of the polychromatic CT forward model and the proposed loss function.\n\n3. The paper is well-written and easy to follow. Experimental details are provided in supplementary materials with the open-sourced code. ", "weaknesses": "1. While the author claims that \"Polyner is superior to its supervised counterparts,\" this is not sufficiently supported by the experimental results and may be caused by unfair comparison.  Directly generalizing pre-trained models to a different size (e.g., 256x256) for testing might not yield optimal results due to variations in noise levels and geometry settings.  A fair comparison should re-train or fine-tune these methods to fit the specific geometry settings. \n\n2. The suboptimal quantitative and visual results of ACDNet and DICDNet, in contrast to their original papers and CNNMAR, might be due to differences in geometry settings.  It is necessary to clarify whether the supervised models were fine-tuned on the IID setting of the DeepLesion dataset.  If not, please provide fine-tuned results (excluding OOD datasets) to ensure an accurate performance evaluation.\n\n3. The color representation of CT images impedes readability.  Presenting grayscale results with an appropriate window to facilitate evaluation is more reasonable.  The goal of MAR is to remove artifacts **and** preserve diagnosis ability. \n\n4. Please compare the proposed method with the state-of-the-art dual-domain methods published in 2022 or 2023.  Additionally, discuss the potential of the proposed method in enhancing traditional dual-domain methods.\n\n5.  Since the proposed method is case-specific and requires optimization for each slice, it is essential to compare iterative reconstruction methods optimized in a similar manner. Moreover, reporting the computational time for a single scan will provide meaningful comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts and valuable comments. Below, we provide point-to-point responses to address your concerns.\n\n---\n\n**Q1. Clarify whether the supervised ACDNet and DICDNet models were fine-tuned on the IID setting of the DeepLesion dataset**\n\n**A1:** The two supervised methods (ACDNet and DICDNet) and one self-supervised method (ADN) are pre-trained on the DeepLesion dataset. However, the size of CT images for the first two is 416×416, but 256×256 for the last one. In our experiments, we uniformly test them on the DeepLesion dataset consisting of images of 256×256 size, called as DeepLesion (256), without conducting any fine-tuning. We acknowledge that this might slightly degrade the performance of ACDNet and DICDNet, weakling our claim “*Polyner is superior to its supervised counterparts*.”\n\nTo address this concern, we construct another DeepLesion dataset consisting of images of 416×416 size, called as DeepLesion (416), *fully following the geometry settings reported in the original papers of ACDNet and DICDNet*. Then, we compare them with our Polyner on the new DeepLesion (416). Note that all the hyper-parameters for optimizing our Polyner are the same as in our original submission, which demonstrates its robustness. \n\nWe show the quantitative results in Table R3. There are three observations: \n- The performances of ACDNet and DICDNet are similar to their original reports on the new DeepLesion (416). For example, ACDNet achieves 40.68 dB and 40.91 dB of PSNR in the original paper and DeepLesion (416), respectively. We thus hold the performance concern caused by geometry settings is resolved. \n- Compared with the DeepLesion (256), the three methods all achieve higher performance on the DeepLesion (416). The reason is the number of projections is 360 and 640 for the DeepLesion datasets (256) and (416), respectively. Increasing the projections can improve reconstruction accuracy. \n- Our method achieves the best reconstruction performance quantitatively. For instance, PSNR improves by 1.04 dB and 1.27 dB compared with ACDNet and DICDNet, respectively. \n\nWe show the qualitative results in Figure R3 in the PDF file of the Global Response. The result of our Polyner is closest to the GT sample. This experiment shows our Polyner can perform compared to the supervised methods for the IID settings of in-domain datasets. \n\nWe will add this experiment for geometry settings into the revised manuscript for a more comprehensive evaluation.\n|\t|DeepLesion (256)|DeepLesion (256)|\tDeepLesion (416)|DeepLesion (416)|\n| :---: |:---: |:---: |:---: |:---: |\n||PSNR|SSIM|PSNR|SSIM|\n|DICDNet|\t37.55±2.52|0.9689±0.0116|\t40.68±1.79|0.9786±0.0061|\n|ACDNet|\t**38.19±2.54**|0.9675±0.0152|\t40.91±2.23|0.9753±0.0120|\n|Polyner (Ours)|\t37.57±1.93|**0.9754±0.0083**|\t**41.95±1.68**|**0.9829±0.0052**|\n\n*Table R3: Quantitative results of DICDNet, ACDNet, and our Polyer on the two versions of DeepLesion datasets.*\n\n---\n\n**Q2. How to compute PSNR and SSIM metrics?**\n\n**A2:** The two metrics are calculated based on the attenuation coefficients.\n\n---\n\n**Q3. Suggest changing the color scheme of the visual results.**\n\n**A3:** In the original submission, we use a color scheme to prominently visualize the differences among reconstructed images. However, we fully agree with Reviewer oo65 that the grayscale scheme is more standard and widely used in CT imaging. To enhance readability, we will replace the color images with a grayscale version in the revised manuscript.\n\n---\n\n**Q4. Supervised dual-domain methods should be compared.**\n\n**A4:** We choose not to include dual-domain MAR methods for two primary reasons:\n- Based on the experimental results reported by the original papers, the SOTA methods ACDNet and DICDNet outperform the well-known dual-domain MAR methods, such as DuDoNet and DuDoNet++. \n- The official code and pre-trained models of the majority of dual-domain methods are not available, making a reliable comparison challenging. \n\nTaking these factors into account, we have decided to use ACDNet and DICDNet as the main deep learning-based comparison methods for our study.\n\n---\n\n**Q5. Some statements are not accurate.**\n\n**A5:** We appreciate the reviewer’s patient proofreading. We will carefully modify these inaccurate statements and typos in the revised submission.\n\n---\n\n**Q6. For 3D reconstruction, Polyner may produce discontinuous z-axis.**\n\n**A6:** As shown in Figure R1 in the PDF file of the Global Responses, we have developed a 3D cone-beam version of Polyner and successfully used it to reconstruct a 3D image of a mouse. The extension of Polyner to support 3D CT imaging is straightforward by modifying the scanning geometry used for simulating the X-rays, and the superior MAR performance is stable. The image continuity along the z-axis is improved in the 3D implementation result.\n\nWe present Polyner in the 2D version to ease a broader comparison with existing MAR works. Due to the training efficiency and memory footprint considerations, most deep-learning MAR models are 2D. A 3D version could limit the fairness of our comparison.\n\n---\n\n**Q7. Comparison with iterative methods and computational time.**\n\n**A7:** Thank you for the suggestion. We have added the algebraic reconstruction technique (ART), a classical iterative CT imaging method, as additional baselines. Table R4 displays the quantitative results. Two observations are evident: \n\n- As iterative methods, both ART and our Polyner are markedly slower than the analytical FBP algorithm. \n- Our Polyner achieves significant improvements compared to the two conventional methods.\n\nWe will include this comparison in the revised manuscript.\n\n||PSNR|SSIM|Average Time|\n| :---:|:---:|:---:|:---:|\n|FBP\t|29.17±3.30|\t0.7231±0.0998|**0.06 s**|\n|ART\t|32.88±3.63|\t0.8352±0.0701|180 s|\n|Poylner (Ours)|**37.57±1.93**|**0.9754±0.0083**|121 s|\n\n*Table R4. Quantitative results of FBP, ART, and our Polyner on the DeepLesion dataset. Note that ART and Polyner are all iterative methods.*\n\n"}, {"review_id": "FrVeUrrwnw", "reviewer": "Reviewer_QLvh", "summary": "The authors tackle a very relevant problem in computed tomography: metal artifact reduction. They recognize some severe gaps in conventional methods (both model-based and deep-learning based) and propose a theoretically sound approach to solve these gaps using implicit neural representations in a somewhat similar fashion as, for example, NeRP. However, to solve the problem at hand, the first derive a polychromatic (as opposed to standard monochromatic) forward model to model the nonlinear CT acquisition process. In combination with a newly developed loss function, they use this model to apply some constraints to the physical properties of polychromatic CT and incorporate the implicit neural representation reconstruction approach. The method is not pre-trained and optimized per scan. They perform better than model-based and supervised deep learning methods on three datasets.", "questions": "1)\tYou assume that you can discretize the energy levels. Is this a valid assumption? Would a continuous approach be possible?\n2)\tWhy have you not evaluated the method on a real-world patient data set?\n3)\tWould it be possible to extend the method to cone-beam CT or other CT acquisition protocols?\n4)\tHow would you make the optimization faster to tackle the weakness regarding computational time?\n5)\tWhy did you opt for ReLU and not SIREN activations, which are continuous, better differentiable, and have shown improved generalization capabilities?\n", "limitations": "The authors mention a sufficient amount of limitations of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I must say that reading this paper was a pleasure. Overall, the manuscript is written very clearly and easy to follow. The ideas are original, and motivated well, and the results are discussed and explained appropriately.\nI think the idea builds nicely upon previous work, aims to tackle a very relevant problem in medical imaging, and applies to a wide range of medical imaging settings. Therefore, the clinical relevance of the work is significant. \nSection 2.1 provides a clear and sound theoretical foundation to understand their approach.\nAll ablation studies are supported by figures that show the importance of the components of the proposed architecture, and the other experiments are done on three different data sets.\nIt is a great strength that the method outperforms or performs equally compared to supervised models without using any training data.\n", "weaknesses": "I am quite familiar with INRs and methods such as NeRP, etc. I feel that the manuscript is complete and does not have severe weaknesses. There are, however, three things that I consider minor weaknesses. Firstly, the paper lacks an evaluation of a real-world patient data set. There are experiments on synthetically induced metal artifacts and the walnut scan. Still, in the end, the method should work on real patient data, and it would be better if the method were evaluated in that setting or if the authors would motivate why this was not done. Secondly, I think 2D fan beam CT is a bit old-fashioned. It would contribute to the strength of the paper – and its applicability in the real world – if the method would also work on cone-beam CT. Lastly, the speed of the approach limits the method significantly. 2 minutes for a 256x256 slice means about 4 minutes for a 512x512 slice. For brain CT, this would take 400 minutes, which is not clinically feasible. I suggest discussing potential methods to optimize the optimization process to make it more relevant for clinical application. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the time and effort the reviewer dedicated to our work. It's genuinely uplifting to know that you consider our work \"*reading this paper was a pleasure*\". Below, we provide point-to-point responses to address your concerns.\n\n---\n\n**Q1. You assume that you can discretize the energy levels. Is this a valid assumption? Would a continuous approach be possible?**\n\n**A1:** As expressed in Eq. 6, the estimated energy-dependent LAC maps are used to compute the projection data acquired with a polychromatic X-ray source. Theoretically, the energy of a polychromatic X-ray covers a continuous range. But in terms of the energy-dependent LAC maps, discretization is necessary due to the limitations of physical instruments and the trade-off between accuracy and computational costs in measurement. Hence, discretizing energy is a common and valid strategy in literature [1][2][3].\n\nThrough our ablation study, it is shown that enhancing the resolution of the energy spectrum improves the MAR performance. To achieve a continuous energy spectrum, leveraging another INR network to learn its continuous representation could be a feasible solution. We would like to extend our method in future work.\n\n> [1] Punnoose, Jacob, et al. \"spektr 3.0—A computational tool for x‐ray spectrum modeling and analysis.\" Medical physics (2016).\n\n> [2] Boone, John M., and J. Anthony Seibert. \"An accurate method for computer‐generating tungsten anode x‐ray spectra from 30 to 140 kV.\" Medical physics (1997).\n\n> [3] Poludniowski, Gavin, et al. \"SpekCalc: a program to calculate photon spectra from tungsten anode x-ray tubes.\" Physics in Medicine & Biology  (2009).\n\n---\n\n**Q2. Why have you not evaluated the method on a real-world patient data set?**\n\n**A2:** There are two factors that hinder our ability to work with clinical patient data:\n\n- The raw projection data and geometry information usually are inaccessible on commercial clinical CT scanners. \n- Using real-world patient data could raise ethical concerns, and it is crucial to handle such data responsibly. \n\nIn this preliminary work, our main objective is to present a novel unsupervised MAR method technically. In the future, we intend to conduct a comprehensive validation of our model using the clinical patient dataset, ensuring a thorough assessment of its performance and efficacy. \n\nHere we additionally scan a mouse thigh containing a metal needle tubing on a commercial Bruker SKYSCAN 1276 micro-CT scanner (Note this data collection is approved ethically). The qualitative results are shown in Figure R1 in the PDF file of the Global Response. Visually, our method still obtains a good reconstruction. We thus believe the efficiency of Polyner has been demonstrated by the walnut and this mouse data with inserted metal.\n\n---\n\n**Q3. Would it be possible to extend the method to cone-beam CT or other CT acquisition protocols?**\n\n**A3:** See Q2 in the Global Responses, please.\n\n---\n\n**Q4. How would you make the optimization faster to tackle the weakness regarding computational time?**\n\n**A4:** See Q1 in the Global Responses, please.\n\n---\n\n**Q5. Why did you opt for ReLU and not SIREN activations, which are continuous, better differentiable, and have shown improved generalization capabilities?**\n\n**A5:** In our Polyner implementation, we leverage hash encoding to enhance the fitting capacity of the MLP network (the experiments about its effectiveness are provided in Q1 of the response to Reviewer ibgY). SIREN is also an effective INR architecture where a good continuous representation can be obtained. \n\nHowever, there are two main reasons we do not use SIREN activations in our Polyner: \n\n- An excellent performance can be achieved by the hash encoding with a simple MLP.\n- Using SIREN activation to replace ReLU will introduce an extra hyper-parameter $\\omega_0$ (See [4] for more details). This might degrade the robustness of our model.\n\nConsidering these factors, we use hash encoding and a simple MLP in our Polyner implementation, as it fulfills our requirements effectively. However, we also believe that exploring a more effective INR architecture is a straightforward and promising way to improve the MAR performance further.\n\n> [4] Sitzmann, Vincent, et al. \"Implicit neural representations with periodic activation functions.\" NeurIPS. 2020."}]

Author rebuttal:
### **Global Responses**

We sincerely thank the reviewers for their insightful comments and suggestions!

We are encouraged by the reviewers' recognition of the novelty (ibgY, KPnC, QLvh), strong motivation (ibgY, oo65, QLvh), technical (oo65, QLvh) and theoretical (QLvh) soundness of our research. Their positive remarks on the anticipated experiments (ibgY, KPnC, QLvh), high performance (ibgY, KPnC, oo65, QLvh), and the clarity of our writing (ibgY, oo65, QLvh) are also highly appreciated.

Here we address two common concerns raised by multiple reviewers. Subsequently, we provide detailed responses to the individual feedback provided by each reviewer.

---

**Q1. Acceleration of optimization process (KPnC, oo65, QLvh)**

**A1:** Polyner uses implicit neural representation (INR) as its core architecture, and in light of the recent advancements in INR, there have been a few methods to speed up image reconstruction during the last two years. In our work, the current Polyner takes about 2 minutes to reconstruct a 2D CT image of 256×256 on a single NVIDIA RTX TITAN GPU.

There are three possible solutions to further accelerate the reconstruction: 
- *Enhanced encoding schemes*. While the hash encoding used in our Polyner implementation currently represents the state-of-the-art (SOTA), exploring more powerful encoding schemes is a straightforward and promising approach.
- *Meta-learning*. An alternative solution involves using meta-learning to facilitate a superior initialization for our Polyner model. Empirically, Tancik et al. [1] demonstrated that meta-learning can significantly accelerate the optimization of INR networks across diverse tasks, including image fitting, sparse-view CT reconstruction, and novel view synthesis. However, this meta-learning-based solution requires an external dataset consisting of numerous artifact-free CT images, which might limit its applicability in certain rare cases. 
- *Specialized acceleration chips*. The proposed Polyner is a deep learning-based method and is training based on general-purpose GPUs. Nevertheless, we could use some specialized acceleration chips to significantly acceleration the optimization.

> [1] Tancik, Matthew, et al. "Learned initializations for optimizing coordinate-based neural representations." CVPR. 2021.

---

**Q2. Extension of our Polyner model to 3D CT imaging (oo65, QLvh)**

**A2:** Our current Polyner model is presented based on 2D fan beam CT. And it can be extended easily for advanced CT acquisition protocols. For instance, in the case of 3D cone-beam settings, only a simple modification following 3D acquisition geometry for X-ray simulation needs to be conducted, while all other steps remain unchanged. One benefit of 3D reconstruction could be the local consistency along the z-axis in the reconstructed image. In addition, there is a potential problem in 3D CT images: The INR network requires a more powerful capacity to represent 3D objects, especially for large image dimensions, which may intensify computational costs, such as memory footprint.

As shown in Figure R1 in the PDF file of the Global Responses, we implement a 3D cone-beam version of the Polyner on a mouse CT scan conducted with a commercial Bruker SKYSCAN 1276 micro-CT scanner (Note this data collection is approved ethically). Our 3D Polyner model was employed to reconstruct a mouse thigh region with a size of 200×200×150. The enlarged figures in Figure R1 illustrate the mouse thigh regions containing metal needle tubing. The whole optimization process requires approximately 10 GB of memory and takes around 32 minutes when executed on a single NVIDIA RTX TITAN GPU. Visually, our method still obtains a good reconstruction, as shown in Figure R1. 

Besides, we prefer to present Polyner in the current 2D fan beam version to facilitate a broader comparison with the recent deep learning-based MAR methods. Due to the training efficiency, most deep-learning MAR models are 2D. A 3D version could limit the fairness of our comparison.

Meta-review: The final score of this paper is SA, A, BA ,BR. Although one reviewer has raised concerns about experimental comparisons, other reviewers found that the proposed method novel and has made sufficient technical contributions. The rebuttal has addressed most of the concerns except for the real-world patient data experiments. The AC has checked the paper, the review comments and the rebuttals, and believe that the paper has made sufficient contributions for the acceptance of NeurIPS. The AC strongly encourages the authors to finish the real-world patient data experiments and incorporate it into the final version of the paper. This acceptance is also based on the condition that the authors promised to released the source codes of this work.
Decision: Accept (poster)