SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines & Metrics. We employ nine representative CT MAR approaches from three categories as baselines: (1) four model-based algorithms (FBP [3], LI [10], NMAR [11], and ART [45]); (2) three supervised CNN-based DL methods (CNN-MAR [7], DICDNet [14], and ACDNet [13]); and (3) two unsupervised DL method (ADN [30] and Score-MAR [9]). To ensure a fair comparison, we evaluate the five DL models using the checkpoints provided by the authors. Among them, CNN-MAR [7] and Score-MAR [9] are respectively trained on the XCOM [44] and LIDC [46] datasets, while the other three methods are trained on the DeepLesion dataset [43]. We use peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) to quantitatively evaluate the MAR performance.
Implementation Details. For our Polymer, we leverage hash encoding [47] in combination with two fully connected (FC) layers of width 128 to implement the MLP network. A ReLU activation function is then applied after the first FC layer. For the hash encoding [47], we configure its hyper-parameters as follow:  $L = 16$ ,  $T = 2^{19}$ ,  $F = 8$ ,  $N_{\mathrm{min}} = 2$ , and  $b = 2$ . To optimize the network, we randomly
Table 1: Quantitative results of compared methods on the DeepLesion [43] and XCOM [44] datasets. The best and second performances are respectively highlighted in **bold** and **underline**.
Category	Method	DeepLesion [43]	XCOM [44]
PSNR	SSIM	PSNR	SSIM
Model-based	FBP [3]	29.98±3.46	0.7470±0.0992	23.76±0.08	0.6506±0.0240
LI [10]	31.85±3.20	0.8483±0.0726	33.57±0.44	0.8866±0.0300
ART [45]	32.88±3.63	0.8352±0.0701	32.45±3.76	0.8322±0.0762
NMAR [11]	32.84±5.07	0.8920±0.0938	36.63±0.23	0.9473±0.0068
Supervised	CNN-MAR [7]	34.22±2.70	0.9240±0.0375	38.52±0.77	0.9591±0.0100
DICDNet [14]	37.55±2.52	0.9689±0.0116	32.65±2.11	0.9449±0.0037
ACDNet [13]	38.19±2.54	0.9675±0.0152	33.04±1.65	0.9343±0.0089
Unsupervised	ADN [30]	33.00±3.21	0.9412±0.0246	25.26±0.94	0.8737±0.0107
Score-MAR [9]	31.66±3.72	0.9174±0.0423	24.92±1.15	0.8754±0.0044
Polymer (Ours)	37.57±1.93	0.9754±0.0083	38.74±0.59	0.9508±0.0001
DeepLesion Dataset (#165)
NMAR
CNN-MAR
DICDNet
XCOM Dataset (#1)
Figure 2: Qualitative results of the compared methods on two samples (#124 and #2) of the DeepLesion [43] and XCOM[44] datasets. The red regions denote metals.
sample 80 X-rays, i.e.,  $|\mathcal{R}| = 80$  in Eqs. (7) & (8), at each training iteration. We set the hyperparameter  $\lambda$  to 0.2 in Eq. (8). We employ the Adam optimizer [48] with default hyper-parameters and set the initial learning rate to 1e-3, which decays by a factor of 0.5 per 1000 epochs. The total number of training epochs is 4000. It is worth noting that all hyper-parameters are tuned on 10 samples from the DeepLesion dataset [43], and are then held constant across all other samples.
Comparison on Simulation Data. Table 1 presents the quantitative results. On the DeepLesion dataset [43], ACDNet [13], our Polymer, and DICDNet [14] achieve the top three performances and significantly outperform other baseline models. Specifically, our Polymer produces the second-best performance, trailing slightly behind ACDNet [13] by only -0.62 dB in PSNR. However, on the XCOM dataset [44], we observe that DICDNet and ACDNet trained on the DeepLesion dataset [43] suffer from severe performance drops due to the OOD problem. Their performance is even lower than
Figure 3: (Left) A Bruker SKYSCAN 1276 micro-CT scanner and a walnut sample with a metal paper clip. (Right) Qualitative results of the compared methods on the walnut sample data. The red regions denote metals.
Score-MAR
that of the model-based NMAR [11] algorithm, with -3.98 dB and -3.59 dB in PSNR, respectively. In contrast, our Polymer obtains the best performance, slightly outperforming CNN-MAR trained on the XCOM dataset [44] by +0.22 dB in PSNR. Fig. 2 shows the qualitative results. Notably, three model-based algorithms (FBP [3], LI [10], and NMAR [11]) and two unsupervised methods (ADN [30] and Score-MAR [9]) cannot yield satisfactory results, exhibiting severe artifacts on both datasets. Conversely, while DICDNet [14] and ACDNet for the DeepLesion dataset [43] yield excellent MAR performances, their results on the XCOM dataset [44] include severe artifacts. This contrasts with CNN-MAR [7], which achieves superior results for the XCOM dataset [44] but not for the DeepLesion dataset [43]. These visual inspections are consistent with the above quantitative comparisons. Remarkably, our proposed Polymer presents clean and fine-detail CT reconstructions for both datasets, indicating its robustness and superiority over existing MAR methods. More visual results can be found in the supplementary material.
Comparison on Real Data. Though our Polymer model performs best on the two simulation datasets, it is important to study its performance on real collected CT data. To achieve this, we insert a metal paper clip into a walnut sample and then scan it with a commercial Bruker SKYSCAN 1276 micro-CT, as shown in Fig 3 (Left). We compare five baselines with our Polymer for reconstructing a 2D slice of the sample. The remaining three baselines (NMAR [11], CNN-MAR [7], and ADN [30]) are not compared because they require prior knowledge of water-bone segmentation, which is invalid for the walnut sample. Fig. 3 (Right) demonstrates the qualitative results. FBP [3] performs the worst, producing severe shadow artifacts. LI [10] exhibits local deformations in the reconstructed images. Score-MAR [9] almost fails due to the domain shift problem. ACDNet [13] is not effective in completely removing these shadow artifacts, it instead produces an offset in image contrast. In comparison, our Polymer recovers the best visual results in both image details and contrast. We also present the quantitative results in Fig. 3. The proposed Polymer achieves the best performance in two metrics. However, it is worth noting that the quantitative performances of all methods are relatively low (the best PSNR of 22 dB by our Polymer) since the reference CT image cannot be considered GT due to the non-negligible non-rigid deformation caused by the metal paper clip insertion. In addition, we compare our Polymer with FDK algorithm [49] on the real mouse tight sample scanned by a 3D cone-beam CT geometry. Related results are provided in the supplementary material.
Influence of Polychromatic CT Forward Model. We explore the efficiency of the polychromatic CT forward model in our Polymer. To this end, we replace it with the linear integral transformation as in [21, 22, 23, 24, 25, 26]. Other model configurations are kept the same for a fair comparison. Fig. 4 demonstrates the qualitative results. From the visualization, it is clear that without the forward model, our Polymer almost fails to handle the shadow artifacts caused by the nonlinear metal effect. In comparison, our full model with the forward model produces a clean CT image that is very close
w/o Poly. CT
w/ Poly. CT
GT
Table 2: Quantitative results of our Polymer ablating the polychromatic CT forward model on the DeepLesion dataset [43].
Module	PSNR
w/o Poly. CT	32.95±4.31
w/ Poly. CT	36.87±1.56
Figure 4: Qualitative results of our Polymer ablating the polychromatic CT forward model on a sample (#6) of the DeepLesion dataset [43]. The red regions denote metals.
Figure 5: Qualitative comparison of the CT images at different energy levels using our Polymer model without (Top Row) and with (Bottom Row) the EDS loss  $\mathcal{L}_{\mathrm{EDS}}$  on a sample (#4) of the DeepLes-sion dataset [43]. The red regions denote metals.
Table 3: Quantitative results of our Polymer model over the EDS loss  $\mathcal{L}_{\mathrm{EDS}}$  on the DeepLesion [43] dataset.
Parameter	PSNR
λ = 0	22.71±4.97
λ = 0.1	33.75±2.30
λ = 0.2	36.87±1.56
λ = 0.3	36.83±1.52
λ = 0.4	36.39±1.79
to the GT image. We present the quantitative results in Table 2. The results indicate that it contributes to an essential improvement of  $+3.92$  dB in PSNR.
Influence of Energy-dependent Smooth Loss. We investigate the influence of the EDS loss on the model performance. To this end, we set different weights  $\lambda \in \{0, 0.1, 0.2, 0.3, 0.4\}$  for the EDS loss in Eq. (9). We show the quantitative results in Table 3. The model performance initially improves with an increasing contribution of the EDS loss but later slightly degrades if its contribution keeps increasing. The best performance is obtained at  $\lambda = 0.2$ . This is because the absence of the EDS loss cannot ensure the smooth changes of the LACs of the body over energy levels. However, excessive bias towards it can result in the forward model degrading as a linear integral transformation, where the polychromatic LACs across all energy levels are identical. Fig. 5 shows the qualitative results. We observe that the CT images at different energy levels randomly change when the EDS loss is not used. In contrast, the regularization can effectively ensure the changes in the LACs are smooth. Moreover, the LACs slowly decrease as the X-ray energy level increases, which is generally consistent with the experimental findings reported in previous works [50].
Influence of Number of Energy Levels. We investigate how the number of discrete energy levels  $N$  affects the model performance. We set the energy range to [20, 120] and uniformly sample the  $N = \{50,76,86,101\}$  energy levels from the range. We also resample the normalized energy spectrum  $\eta \in \mathbb{R}^N$  to match these levels. Table 4 shows the quantitative results, which indicate that increasing the energy resolution improves the MAR performance. We present the qualitative results in Fig. 6, which show that our Polymer with a higher energy resolution ( $N = 101$ ) produces better CT results in terms of both image details and contrast. Our method approximates polychromatic X-rays with discrete energy levels, introducing an approximation error. Therefore, improving the energy's resolution may help to reduce this error and improve the model performance.
5 Conclusion & Limitation
We present Polyner, a novel method for the nonlinear MAR problem. The proposed Polyner follows the unsupervised learning paradigm and does not require any external training data, which greatly makes it useful in clinical scenarios. Our Polyner learns a neural representation of polychromatic
Figure 6: Qualitative results of our Polymer ablating the discrete energy levels  $N$  on a sample (#9) of the DeepLesion dataset [43]. The red regions denote metals.
Table 4: Quantitative results of our Polymer ablating the discrete energy levels  $N$  on the DeepLesion [43] dataset.
#Energy Levels	PSNR
N = 50	30.98±1.02
N = 76	34.99±1.30
N = 86	36.15±1.38
N = 101	36.87±1.56
CT images to fundamentally avoid the nonlinear metal effect causing metal artifacts. It thus can reconstruct clean CT images from metal- affect measurements. Extensive experiments on three CT datasets demonstrate that our Polyner yields comparable or even better supervised DL methods.
Though the proposed Polyner achieves excellent MAR performance, there still are some limitations. Firstly, our Polyner is case-specific, meaning an independent model has to be optimized for each sample. Technically, the optimization of the Polyner for a CT image of  $256 \times 256$  size requires about 2 minutes on a single NVIDIA RTX TITAN GPU (24 GB). In addition, our current Polyner model is based on 2D fan-beam and 3D cone-beam CT, while more advanced types of beams (e.g., helical CT) are not implemented.
This work was supported by the National Natural Science Foundation of China under Grants No. 62071299 and MoE Key Lab of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University). Ce Wang was supported by the National Natural Science Foundation of China under Grants No. 62301532, in part by the Natural Science Foundation of Jiangsu Province under Grant No. BK20230282
[1] Ge Wang, Hengyong Yu, and Bruno De Man. An outlook on x-ray ct research and development. Medical physics, 35(3):1051-1064, 2008.
[2] Jin Keun Seo and Eung Je Woo. Nonlinear inverse problems in imaging. John Wiley & Sons, 2012.
[3] Avinash C Kak and Malcolm Slaney. Principles of computerized tomographic imaging. SIAM, 2001.
[4] Lars Gjesteby, Bruno De Man, Yannan Jin, Harald Paganetti, Joost Verburg, Drosoula Giantsoudi, and Ge Wang. Metal artifact reduction in ct: where are we after four decades? *IEEE Access*, 4:5826-5849, 2016.
[5] Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-268, 1992.
[6] Muhammad Usman Ghani and W Clem Karl. Fast enhanced ct metal artifact reduction using data domain deep learning. IEEE Transactions on Computational Imaging, 6:181-193, 2019.
[7] Yanbo Zhang and Hengyong Yu. Convolutional neural network based metal artifact reduction in x-ray computed tomography. IEEE transactions on medical imaging, 37(6):1370-1381, 2018.
[8] Wei-An Lin, Haofu Liao, Cheng Peng, Xiaohang Sun, Jingdan Zhang, Jiebo Luo, Rama Chellappa, and Shaohua Kevin Zhou. Dudonet: Dual domain network for ct metal artifact reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10512-10521, 2019.
[42] J Punnoose, J Xu, A Sisniega, W Zbijewski, and JH Siewerdsen. Technical note: spektr 3.0—a computational tool for x-ray spectrum. Medical Physics, 43(8):4711-4717, 2016.
[43] Ke Yan, Xiaosong Wang, Le Lu, Ling Zhang, Adam P Harrison, Mohammadhadi Bagheri, and Ronald M Summers. Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9261-9270, 2018.
[44] Martin J Berger. Xcom: Photon cross sections database, version 1.4. http://physics.nist.gov/xcom, 2009.
[45] Richard Gordon, Robert Bender, and Gabor T Herman. Algebraic reconstruction techniques (art) for three-dimensional electron microscopy and x-ray photography. Journal of theoretical Biology, 29(3):471-481, 1970.
[46] Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics, 38(2):915–931, 2011.
[47] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4), jul 2022.
[48] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
[49] Lee A Feldkamp, Lloyd C Davis, and James W Kress. Practical cone-beam algorithm. Josa a, 1(6):612-619, 1984.
[50] RC Chen, Renata Longo, Luigi Rigon, Fabrizio Zanconati, Alessandro De Pellegrin, Fulvia Arfelli, Diego Dreossi, RH Menk, E Vallazza, TQ Xiao, et al. Measurement of the linear attenuation coefficients of breast tissues by synchrotron radiation computed tomography. Physics in Medicine & Biology, 55(17):4993, 2010.
Supplementary Material—Unsupervised Polychromatic Neural Representation for CT Metal Artifact Reduction
Figure 7: Ten shapes of metals for synthesizing metal-corrupted measurements in the DeepLesion [43] dataset. These metals are supposed as Titanium. The white regions denote metals.
Figure 8: (Left) The simulated spectrum within an energy range of [20, 120] for synthesizing metal-corrupted measurements of the DeepLesion [43] and XCOM [44] datasets. The spectrums estimated within the energy ranges of [0, 100] and [0, 60] estimated by the SPEKTR toolkit [42] for the real walnut sample (Middle) and mouse thigh (Right).
A Additional Details of Datasets
In this work, we perform experiments on four datasets: DeepLesion [43], XCOM [44], and our Walnut Sample. DeepLesion and XCOM are simulation datasets, while Walnut Sample and Mouse Thigh are real-world datasets. Note that our method is fully unsupervised, and thus all the data are exclusively used for testing purposes.
DeepLesion. To simulate metal-corrupted measurements in the DeepLesion dataset [43], we follow the pipelines outlined in [30, 7, 8]. For our experiments, we extract 200 2D images from the DeepLesion dataset as test GT samples [43]. As depicted in Fig. 7, we leverage ten shapes of metallic implants from [30, 7, 8] and consider them Titanium. To simulate the X-ray source, we employ a polychromatic X-ray with an energy range of [20, 120] KeV and a minimum energy unit of  $1\mathrm{KeV}$ . The number of photons emitted by the X-ray source is set to  $2\times 10^{7}$ . The normalized energy spectrum  $\eta$  of the X-ray source is illustrated in Fig. 8 (Left). We adopt an equiangular fan-beam CT acquisition geometry, and the detailed parameters are provided in Table 5. Additionally, we incorporate Poisson noise and consider the partial volume effect in the sinogram domain during the simulation process.
XCOM. For the XCOM dataset [44], we use two samples provided by Zhang et al. [7]. These two cases are simulated using two 2D clean CT images sourced from the XCOM [44] database. Zhang et
Table 5: Detailed parameters of the acquisition geometry for the four datasets.
Parameters	Simulation Datasets	Real Datasets
DeepLesion [43]	XCOM [44]	Walnut	Mouse Thigh
Type of geometry	2D fan-beam	2D fan-beam	2D fan-beam	3D cone-beam
Source Voltage (kV)	120	120	100	60
Source Current (uA)	-	-	200	200
Exposure Time (ms)	-	-	276	730
Image size	256×256	512×512	650×650	200×200×150
Voxel size (mm)	1×1	0.8×0.8	0.05×0.05	0.06×0.06×0.06
Angle range (°)	[0, 360)	[0, 360)	[0, 360)	[0, 360)
The number of the angles	360	984	720	900
Angular spacing (°)	0.1	0.057	-	-
Detector spacing (mm)	-	-	0.069	0.069
Distance from source to center (mm)	362	743	92.602	92.602
Distance from center to detector (mm)	362	743	65.946	65.946
al. [7] consider a polychromatic X-ray source with an energy range of [20, 120] KeV and a minimum energy unit of  $1\mathrm{KeV}$ . The corresponding normalized energy spectrum  $\eta$  is depicted in Fig. 8 (Left). In order to generate metal-corrupted sinograms, an equiangular fan-beam CT acquisition geometry is employed, and the geometry parameters are specified in Table 5. Similar to the DeepLesion [43] dataset, Zhang et al. [7] also simulate Poisson noise and consider the partial volume effect in the sinogram domain during the simulation process.
Walnut Sample. To assess the performance of our proposed method on real CT data, we employ a commercial Bruker SKYSCAN 1276 micro-CT scanner to scan a walnut sample that contains a metal paper clip. Detailed parameters of the acquisition geometry can be found in Table 5. To estimate the X-ray spectrum of the micro-CT scanner, we leverage the SPEKTR toolkit [42]. The estimated spectrum is illustrated in Fig. 8 (Middle).
Mouse Thigh. We also scan a mouse thigh containing a metal intramedullary needle on the same micro-CT scanner. This sample is 3D cone-beam data. Detailed parameters of the acquisition geometry are shown in Table 5. We leverage the SPEKTR toolkit [42] to estimate the X-ray spectrum of the micro-CT scanner. The estimated spectrum is illustrated in Fig. 8 (Right).
B Additional Details of Baselines
In our experiments, we compare our proposed method against eight baseline MAR approaches. Notably, for the five DL-based methods, we evaluate their performance using the pre-trained models provided by the respective authors.
FBP. The FBP [3] is a conventional approach used for reconstructing CT images. It involves re-projecting the acquired sinogram data onto the image domain using the corresponding projection angles and geometric parameters to obtain an approximate estimate of the unknown image. In our experiments, we use the in-build function ifanbem in MATLAB (https://ww2.mathworks.cn/help/images/ref/ifanbeam.html?requestedDomain  $\equiv$  cn).
LI. A simple strategy to mitigate metal artifacts in CT imaging involves the direct linear interpolation of the sinogram [10] to fill in the regions affected by metal. This approach does not require any network training but may result in imperfect sinogram completion, which in turn could introduce undesired artifacts in the reconstructed image. In our experiments, we use the implementation provided by Zhang et al. [7] (https://github.com/yanbozhang007/CNN-MAR/blob/master/cnnmar).
NMAR. The NMAR [11] introduces a generalized normalization technique that extends previously developed interpolation-based MAR techniques (e.g., LI [10]), which also does not require any network training. Specifically, it normalizes the projections before interpolation based on forward projections of a prior image obtained through multi-threshold segmentation of the initial image. In our experiments, we use the implementation provided by Zhang et al. [7] (https://github.com/yanbozhang007/CNN-MAR/tree/master/cnnmar).
CNN-MAR. Zhang et al. [7] proposed a CNN-based MAR framework, which uses a CNN to estimate a prior image and subsequently apply a sinogram correction. However, despite the strong representation ability of CNNs, these approaches are still susceptible to secondary artifacts resulting from inconsistent sinograms. In our experiments, we use its official implementation and pre-trained model (https://github.com/yanbozhang007/CNN-MAR/tree/master/cnnmar).
DICDNet. Wang et al. [14] propose a deep interpretable convolutional dictionary network (DICDNet) for the MAR task, which explicitly formulates the prior structures underlying metal artifacts in CT images as a convolutional dictionary model. In our experiments, we use its official implementation and pre-trained model (https://github.com/hongwang01/DICDNet/tree/main).
ACDNet. Similarly to DICDNet [14], the adaptive convolutional dictionary network (ACDNet) [13] explicitly encodes the prior observations underlying the MAR task into an adaptive convolutional dictionary network. In our experiments, we use its official implementation and pre-trained model (https://github.com/hongwang01/ACDNet).
ADN. The ADN [30] is an unsupervised learning approach for the MAR problem. Specifically, it takes unpaired metal-corrupted and clean CT images as inputs to learn the transformation between these two distributions. In our experiments, we use its official implementation and pre-trained model (https://github.com/liaohaofu/adn/tree/master).
Score-MAR. Song et al. [9] demonstrated that unconditional diffusion models can be adapted to various inverse problems, such as the MAR. Specifically, it learns the prior distribution of metal-free CT images with a generative model in order to infer the lost sinogram in the metal-affected regions. In our experiments, we use its official implementation and pre-trained model (https://github.com/yang-song/score_inverse/problems).
C Additional Visual Results
Figs. 9, 10, 11, 12, and 13 demonstrate some additional visual results. We observe that the proposed Polymer generally obtains the best MAR results.
Our Polyner is expected to have significant broader impacts in the field of medical imaging. By effectively reducing metal artifacts in CT scans, our research has the potential to improve diagnostic accuracy, leading to more precise diagnoses and enhanced patient care. However, it is important to address potential limitations and concerns, such as the possibility of introducing false positive or negative results. Thorough evaluation and validation of our method is crucial to ensure its reliability and minimize any adverse effects. Overall, our work contributes to advancing medical imaging technology and has the potential to positively impact healthcare.
FEDK
Figure 9: (Left) A sample among 2D projections of a mouse thigh containing a metal intramedullary needle scanned by the micro-CT scanner. (Right) Qualitative results of FDK [49] and our Polymer on the sample. Note that the acquisition geometry is the 3D cone beam. The reconstructed images have a size of  $200 \times 200 \times 150$ . Our Polymer takes about 32 minutes on a single NVIDIA RTX TITAN GPU. The red regions denote the metal needle tubing. This data collection is approved ethically.
ACDNet