SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qing Wu\* Lixuan Chen\* Ce Wang\* Hongjiang Wei\* S. Kevin Zhou\* Jingyi Yu\* Yuyao Zhang\*,*
\*ShanghaiTech University
$\hat{\mathcal{O}}$  Shanghai Jiao Tong University
$\spadesuit$ University of Science and Technology of China
$\diamond$ Institute of Computing Technology, Chinese Academy of Sciences
{wuqing, chenlx1, yujingyi, zhangyy8}@shanghaiitech.edu.cn
wangce@ict.ac.cn hongjiang.wei@sjtu.edu.cn skevinzhou@ustc.edu.cn
Emerging neural reconstruction techniques based on tomography (e.g., NeRF, NeAT, and NeRP) have started showing unique capabilities in medical imaging. In this work, we present a novel Polychromatic neural representation (Polyner) to tackle the challenging problem of CT imaging when metallic implants exist within the human body. CT metal artifacts arise from the drastic variation of metal's attenuation coefficients at various energy levels of the X-ray spectrum, leading to a nonlinear metal effect in CT measurements. Recovering CT images from metal-affected measurements hence poses a complicated nonlinear inverse problem where empirical models adopted in previous metal artifact reduction (MAR) approaches lead to signal loss and strongly aliased reconstructions. Polymer instead models the MAR problem from a nonlinear inverse problem perspective. Specifically, we first derive a polychromatic forward model to accurately simulate the nonlinear CT acquisition process. Then, we incorporate our forward model into the implicit neural representation to accomplish reconstruction. Lastly, we adopt a regularizer to preserve the physical properties of the CT images across different energy levels while effectively constraining the solution space. Our Polymer is an unsupervised method and does not require any external training data. Experimenting with multiple datasets shows that our Polymer achieves comparable or better performance than supervised methods on in-domain datasets while demonstrating significant performance improvements on out-of-domain datasets. To the best of our knowledge, our Polymer is the first unsupervised MAR method that outperforms its supervised counterparts. The code for this work is available at: https://github.com/iwuqing/Polyner.
Computed tomography (CT) is a widely-used imaging technique for clinical diagnosis. CT measurements are produced by passing X-rays at different angles through the human body. Reconstruction of CT images from these measurements is often approximately modeled as a linear inverse problem [1]. However, the attenuation coefficients of the metal materials dramatically vary with the X-ray spectral energy, which largely differs from those of human body tissues. This results in a non-negligible nonlinear metal effect in the CT measurements when the body carries metallic implants [2]. Hence,
recovering CT images from metal-cause measurements is a complicated nonlinear inverse problem. However, linear inverse problems, e.g., filtered back projection (FBP) [3], always lead to severe artifacts in the resulting CT images. These artifacts can greatly hamper the accuracy of clinical diagnosis and treatment planning.
Many efforts have been made for CT metal artifact reduction (MAR) task [4]. Most studies [5, 6, 7, 8, 9, 10, 11, 12] simplify this task as a linear inverse problem by designing various schemes to replace metal-affected extreme value signals (i.e., metal traces) in the measurements, which unfortunately results in significant degradation of the CT reconstruction due to the signal loss. Currently, supervised deep learning (DL) approaches [7, 8, 13, 14, 15, 16, 17, 6] are the mainstream MAR solutions. Supervised methods typically tend to learn mappings from metal-corrupted measurements to artifact-free CT images by training deep neural networks on a large-scale external dataset. However, related works face two main limitations. First, collecting numerous paired metal-corrupted measurements and high-quality CT images for training is resource-intensive and laborious. Second, if the metal shapes and the CT acquisition settings differ from those in the training dataset, performance may significantly decline. These challenges significantly limit the practical application of supervised MAR methods in clinical scenarios.
Implicit neural representation (INR) is a new paradigm to solve inverse problems [18]. It uses a multi-layer perception (MLP) to model an underlying signal by mapping coordinates to signal responses. The learning bias of the MLPs towards low-frequency signal components [19, 20] is considered as an implicit regularization term for the inverse problems. By approximately simulating the CT acquisition forward model as a linear integral, INR has achieved significant progress in the linear sparse-view CT reconstruction [21, 22, 23, 24, 25, 26]. However, its potential for tackling the nonlinear inverse problem, such as the MAR task, remains unexplored.
In this work, we aim to address the issue of MAR in CT scans from a nonlinear perspective. This allows us to use the complete measurement signals (i.e., including metal traces) and thus achieve better CT reconstruction performance. Specifically, we model the MAR as a polychromatic (i.e., multiple X-ray spectral energy levels) reconstruction task of CT image. This task is a highly ill-posed nonlinear inverse problem. To address this, we propose Polychromatic neural representation (Polyner), a novel extension of INR for the nonlinear problems. In our Polyner, we make three key designs. First, we derive a forward model that accurately simulates the polychromatic nonlinear CT acquisition process. Second, we define a regularization term that effectively constrains the solution space by preserving the physical properties of the CT images across different spectral energy levels. Third, we incorporate our forward model into the INR, granting its capacity to solve the nonlinear inverse problem. Meanwhile, the continuous prior brought by INR also contributes to a more stable solution. Our Polyner is an unsupervised method and does not require any external training data, which makes it potentially applicable to a wide range of CT imaging scenarios.
We evaluate the performance of our Polyner on three datasets, including two simulation datasets and one real-world collected dataset. The experimental results demonstrate that our Polyner performs comparably to the supervised DL method on in-domain datasets, while significantly outperforming them on out-of-domain datasets. We also conduct extensive ablation studies for several key designs of our method, verifying their effectiveness. To the best of our knowledge, our Polyner is the first unsupervised MAR method superior to its supervised counterparts.
The main contributions of this work can be summarised as follows: (1) We propose Polyner, an unsupervised polychromatic neural representation, that can recover high-quality CT images from metal-corrupted measurements without the need for any extra data. (2) We introduce a new model perspective for the CT MAR problem, i.e., recovering polychromatic CT images to mitigate the nonlinear metal effect and thus achieving effective MAR. (3) We technically incorporate a polychromatic CT forward model into the INR, enabling the reconstruction of polychromatic CT images.
2 Preliminaries & Related Work
In this section, we first theoretically present the nonlinear metal effect during the CT acquisition process (Sec. §2.1). Then, we provide a brief overview of the literature on the metal artifacts reduction approaches (Sec. §2.2) and implicit neural representations for CT reconstruction (Sec. §2.3).
2.1 Nonlinear Metal Effect in CT Measurements
The physical interaction of X-rays passing through an object is governed by Lambert-Beer's law [27, 28], which can be expressed as follows:
$$
I (\mathbf {r}, E _ {0}) = I _ {0} (E _ {0}) \cdot \exp \left(- \int_ {\mathbf {r}} \mu_ {E _ {0}} (\mathbf {x}) \mathrm {d} \mathbf {x}\right), \tag {1}
$$
where  $\mathbf{r}$  denotes a monochromatic X-ray,  $E_0$  is its energy level,  $I_0(E_0)$  is the number of photons emitted by an X-ray source,  $I(\mathbf{r}, E_0)$  is the number of photons received by an X-ray detector, and  $\mu_{E_0}(\mathbf{x})$  is the linear attenuation coefficient (LAC) of the observed object at the position  $\mathbf{x}$ . The LACs serve as a measure of the object's ability to absorb the X-rays, with higher values indicating greater absorption ability.
In practice, X-rays often are polychromatic due to limitations in the X-ray source [2, 29]. Consider a polychromatic X-ray composed of multiple monochromatic X-rays belonging to an energy level range  $\mathcal{E}$ , by combining Eq. (1), the CT measurement data  $p(\mathbf{r})$  thus can be written as below:
$$
\begin{array}{l} p (\mathbf {r}) = - \ln \frac {\int_ {\mathcal {E}} I (\mathbf {r} , E) \mathrm {d} E}{\int_ {\mathcal {E}} I _ {0} (E) \mathrm {d} E} \tag {2} \\ = - \ln \int_ {\mathcal {E}} \eta (E) \cdot \exp \left(- \int_ {\mathbf {r}} \mu_ {E} (\mathbf {x}) \mathrm {d} \mathbf {x}\right) \mathrm {d} E, \text {w i t h} \eta (E) = \frac {I _ {0} (E)}{\int_ {\mathcal {E}} I _ {0} (E ^ {\prime}) \mathrm {d} E ^ {\prime}}, \\ \end{array}
$$
where  $\eta (E)$  is the normalized energy spectrum that characterizes the distribution of the number of photons  $I_0(E)$  emitted by the X-ray source over the energy level  $E$ .
CT reconstruction refers to solve the LACs  $\mu$  from the measurement data  $p$ . It is widely recognized that the LACs of human body tissues exhibit slow decreases with increasing the X-ray's energy level, while those of metals undergo sharp variations as a function of energy level [2, 7, 8, 30]. Therefore, a usual assumption for the human body is that  $|\mu_{E_a}(\mathbf{x}) - \mu_{E_b}(\mathbf{x})| \approx 0$ ,  $\forall E_a, E_b \in \mathcal{E}$ , i.e., the LACs of the human body are considered as energy-independent.
Based on this assumption, we can represent the underlying LACs of a human body with metallic implants as follows:
$$
\mu_ {E} (\mathbf {x}) = \mathcal {M} (\mathbf {x}) \cdot \mu_ {E} ^ {\star} (\mathbf {x}) + [ 1 - \mathcal {M} (\mathbf {x}) ] \cdot \mu (\mathbf {x}), \tag {3}
$$
where  $\mu_{E}^{\star}(\mathbf{x})$  and  $\mu (\mathbf{x})$  denote the LACs of the metal and human body, respectively.  $\mathcal{M}$  is a metal binary mask  $(\mathcal{M} = 1$  for the metal region and  $\mathcal{M} = 0$  otherwise).
By combining Eq. (2), its CT measurement data can be derived as:
$$
\begin{array}{l} p (\mathbf {r}) = - \ln \int_ {\mathcal {E}} \eta (E) \cdot \exp \left(- \int_ {\mathbf {r}} \mathcal {M} (\mathbf {x}) \cdot \mu_ {E} ^ {\star} (\mathbf {x}) + [ 1 - \mathcal {M} (\mathbf {x}) ] \cdot \mu (\mathbf {x}) d \mathbf {x}\right) d E \\ = \underbrace {- \ln \int_ {\mathcal {E}} \eta (E) \cdot \exp \left(- \int_ {\mathbf {r}} \mathcal {M} (\mathbf {x}) \cdot \mu_ {E} ^ {\star} (\mathbf {x}) \mathrm {d} \mathbf {x}\right) \mathrm {d} E} _ {\text {N o n l i n e a r M e t a l E f f e c t}} + \underbrace {\int_ {\mathbf {r}} [ 1 - \mathcal {M} (\mathbf {x}) ] \cdot \mu (\mathbf {x}) \mathrm {d} \mathbf {x}} _ {\text {L i n e a r I n t e g r a l}}. \tag {4} \\ \end{array}
$$
This implies that the CT measurement of a human body with metallic implants can be divided into two parts: (1) a linear integral transformation for the human body and (2) a nonlinear transformation for the metallic implants. The latter is also known as the nonlinear metal effect in CT measurements. Therefore, using standard reconstruction techniques, such as FBP [3], which are designed for linear inverse problems, will result in severe metal artifacts in the reconstructed CT image.
2.2 CT Metal Artifact Reduction Approaches
Classical model-based algorithms [10, 11, 12] formulate the MAR as an image inpainting task. The metal traces in the measurements are treated as missing and completed by interpolation. However, these model-based methods often produce severe secondary artifacts since the interpolation algorithms ignore the CT physical geometry constraints. With the great success of deep neural networks in low-level vision tasks [31, 32], numerous supervised DL methods have been proposed [7, 8, 13, 14, 6, 15, 17, 16]. For example, Lin et al. [8] proposed a dual domain network (DuDoNet) to
Figure 1: Overview of the proposed Polyner model. Firstly, we sample coordinates  $\mathbf{x}$  along an X-ray  $\mathbf{r}$ . Then, we feed these coordinates into an MLP to predict the corresponding LACs  $\mathbf{v}(\mathbf{x}) = \{\mu_1(\mathbf{x}),\dots ,\mu_N(\mathbf{x})\}$  at  $N$  energy levels. Thirdly, we leverage a differentiable polychromatic CT forward model to transform the predicted LACs into CT measurements  $\hat{p} (\mathbf{r})$ . Finally, we optimize the MLP by minimizing the loss between the predicted and real CT measurements.
simultaneously complete the measurements and enhance the CT images. Wang et al. [13] presented an adaptive convolutional dictionary network (ACDNet), incorporating prior knowledge of metal artifacts into a deep network. These supervised methods achieve state-of-the-art (SOTA) MAR performance due to their well-designed network architectures and data-driven priors. However, they require a large-scale training dataset for supervised learning and often suffer from out-of-domain (OOD) problems [9, 30], severely limiting their usefulness in real-world scenarios. In contrast, our Polymer is a fully unsupervised method that does not require any external training data. Moreover, both model-based and supervised DL methods often treat metal traces as missing data, which severely hinders the restoration of image details near metals. Our method decomposes the CT measurement with polychromatic LACs at different energy levels as in Eq. (2), and directly utilizes the complete measurement as input data, which avoids signal loss of the metal traces and thus achieves a better MAR performance.
2.3 Implicit Neural Representation for CT Reconstruction
Implicit neural representation (INR) is a novel paradigm to continuously parameterize a variety of signals. INR represents an underlying signal as a continuous function that maps spatial positions to signal responses and uses an MLP network to approximate the complex function. Due to the learning bias of the MLP networks to low-frequency signals [19, 20], INR can be used for various vision inverse problems, e.g., view synthesis [33, 34], and surface reconstruction [35, 36]. Recently, many INR-based approaches [22, 23, 24, 25, 37, 21, 38, 39, 40, 26, 41] have been emerged for CT reconstruction. There are two key designs in these methods: (1) representing the unsolved object as a function that maps spatial coordinates to its corresponding monochromatic LCAs at a single energy level, and (2) combining line integral transformation with an MLP network to approximate the function. They have shown excellent CT reconstruction performance benefiting from implicit priors by INR. However, these methods fail to handle the nonlinear CT MAR task because they strictly suppose the unsolved LACs are monochromatic, i.e., energy-independent, leading to the nonlinear metal effect being ignored. In contrast, our method leverages a polychromatic forward model to accurately describe the formation of the metal effect and thus enables an effective MAR.
In this section, we propose our Polymer model. We first introduce our modeling for the CT MAR problem (Sec. §3.1). Next, we present a polychromatic CT forward model for simulating the CT acquisition process (Sec. §3.2). Moreover, we define the loss function to optimize our Polymer (Sec. §3.3). Finally, we provide the pipeline to recover the reconstruction results (Sec. §3.4). An overview of the proposed Polymer model is shown in Fig. 1.
The LACs of metals to X-rays vary significantly with the spectral energy level of the X-ray, resulting in metal- affect extreme signal values in the measurement data (i.e., nonlinear metal effect discussed in
Sec. §2.1). Recovering the underlying CT image from such measurement is a complicated nonlinear inverse problem. Existing MAR approaches [5, 6, 7, 8, 9, 10, 11, 12] mostly formulated it as a linear inverse problem by removing the metal-affected parts in the measurement.
Instead, we aim to address the MAR problem from a nonlinear perspective. We suppose that the polychromatic X-rays in the CT acquisition can be decomposed into monochromatic X-rays at  $N$  discrete energy levels. Then, we propose to reconstruct the polychromatic CT images at each of the  $N$  energy levels (i.e., a total number of  $N$  LAC-maps at each monochromatic X-ray for the observed object). This reconstruction task is nonlinear and is independent of the nonlinear metal effect. Hence, the MAR task is modeled as a reconstruction problem of the polychromatic CT images.
To accomplish the reconstruction, we represent the underlying object as a continuous function of spatial coordinate, which can be expressed as below:
$$
f: \mathbf {x} = (x, y) \in \mathbb {R} ^ {2} \longrightarrow \mathbf {v} (\mathbf {x}) = \left\{\mu_ {1} (\mathbf {x}), \dots , \mu_ {i} (\mathbf {x}), \dots , \mu_ {N} (\mathbf {x}) \right\} \in \mathbb {R} ^ {N}, \tag {5}
$$
where  $\mathbf{x}$  denotes any spatial coordinate and  $\mathbf{v}(\mathbf{x})$  is the polychromatic CT image value at that position. Here the item  $\mu_{i}(\mathbf{x})$  is the LAC of the object to the monochromatic X-ray at energy level  $E_{i}$ .
However, the function  $f$  is very complicated and intractable. Hence, we leverage an MLP network  $\mathcal{F}_{\Phi}$  (here  $\Phi$  denotes trainable weights) to approximate it. In other words, we learn a neural representation of the function  $f$ . The artifacts-free CT images can be reconstructed by feeding all the spatial coordinates into the well-optimized MLP network  $\mathcal{F}_{\Phi}$ .
3.2 Polychromatic CT Forward Model
To learn the function  $f$ , we optimize the MLP network's trainable weights  $\Phi$  to map any spatial coordinate  $\mathbf{x}$  in space into its corresponding polychromatic LACs  $\mathbf{v}(\mathbf{x})$ . Therefore, a differentiable forward model is required for transforming the LACs predicted by the MLP network  $\mathcal{F}_{\Phi}$  into the CT measurements while also allowing the back-propagation of gradients from the measurement domain to the image domain.
We derive a polychromatic forward model for X-ray CT systems in Eq. (2). This forward model can accurately describe the complicated nonlinear CT acquisition process. Our Polyner employs its discrete form to transform the MLP-predicted polychromatic LACs  $\mathbf{v}(\mathbf{x})$  into the measurement data  $\hat{p}$ . Fig. 1 illustrates its pipeline. For the LACs  $\mu_{i}$  at the energy level  $E_{i}$ , we first generate its projection value  $\hat{p}_i(\mathbf{r})$  along the X-ray  $\mathbf{r}$  using a linear summation operator. We then leverage the normalized energy spectrum  $\eta(E_i)$  to weight the sum of  $\hat{p}_i(\mathbf{r})$  and produce the final measurement data  $\hat{p}(\mathbf{r})$ . Formally, the polychromatic CT forward model can be expressed as follows:
$$
\hat {p} (\mathbf {r}) = - \ln \sum_ {i = 1} ^ {N} \eta \left(E _ {i}\right) \cdot \exp \left\{- \hat {p} _ {i} (\mathbf {r}) \right\}, \text {w i t h} \hat {p} _ {i} (\mathbf {r}) = \sum_ {\mathbf {x} \in \mathbf {r}} \mu_ {i} (\mathbf {x}) \cdot \Delta \mathbf {x}, \tag {6}
$$
where  $\Delta \mathbf{x}$  represents the distance between adjacent sampled coordinates along the X-ray  $\mathbf{r}$ . It is set as the physical resolution (i.e., voxel size) of the reconstructed CT image. Here, the normalized energy spectrum  $\eta \in \mathbb{R}^N$  is considered as a known prior knowledge. In our experiments, we leverage SPEKTR toolkit developed by Punnoose et al. [42] to estimate it.
3.3 Loss Function
Data Consistency Loss. We first compute a data consistency (DC) loss that measures the distance between the predicted and measured measurements. The DC loss is implemented by  $\ell_1$  norm, which can be written as below:
$$
\mathcal {L} _ {\mathrm {D C}} = \frac {1}{| \mathcal {R} |} \sum_ {\mathbf {r} \in \mathcal {R}} \| p (\mathbf {r}) - \hat {p} (\mathbf {r}) \| _ {1}, \tag {7}
$$
where  $\mathcal{R}$  denotes a set of the sampling X-rays  $\mathbf{r}$  at each training iteration.
Energy-dependent Smooth Loss. As discussed in Sec. §2.1, the LACs of the human body tissues to X-rays exhibit smooth changes with increasing the energy level of the X-rays [7, 8, 30]. We thus propose an energy-dependent smooth (EDS) loss to preserve this physical property. Specifically, for the human body, the proposed EDS loss enforces a regularization that constrains the changes between the LACs at any adjacent energy levels to be smooth. We implement this regularization using the  $\ell_1$
norm of the gradient along the energy spectrum. It is approximated by the sum of the absolute error between the predicted LACs at any adjacent X-ray energy levels. Mathematically, the EDS loss can be written as follows:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {E D S}} = \frac {1}{| \mathcal {R} | \cdot | \mathbf {r} |} \sum_ {\mathbf {r} \in \mathcal {R}} \sum_ {\mathbf {x} \in \mathbf {r}} [ 1 - \mathcal {M} (\mathbf {x}) ] \cdot \| \nabla_ {E} \mathbf {v} (\mathbf {x}) \| _ {1} \tag {8} \\ = \frac {1}{| \mathcal {R} | \cdot | \mathbf {r} |} \sum_ {\mathbf {r} \in \mathcal {R}} \sum_ {\mathbf {x} \in \mathbf {r}} [ 1 - \mathcal {M} (\mathbf {x}) ] \cdot \sum_ {i = 2} ^ {N} | \mu_ {i - 1} (\mathbf {x}) - \mu_ {i} (\mathbf {x}) |, \\ \end{array}
$$
where  $\mathcal{M}$  is a metal binary mask ( $\mathcal{M} = 1$  for the metal region and  $\mathcal{M} = 0$  otherwise).
In summary, our total loss function is defined as below:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {D C}} + \lambda \cdot \mathcal {L} _ {\mathrm {E D S}}, \tag {9}
$$
where  $\lambda \geq 0$  is a hyper-parameter that controls the contribution of the EDS regularization  $\mathcal{L}_{\mathrm{EDS}}$ .
3.4 Reconstruction of Artifact-free CT Image
Once the optimization is completed, the MLP network  $\mathcal{F}_{\Phi}$  is able to predict the polychromatic LACs  $\mathbf{v}(\mathbf{x}) = \{\mu_1(\mathbf{x}),\dots ,\mu_i(\mathbf{x}),\dots ,\mu_N(\mathbf{x})\}$  at  $N$  energy levels for any input spatial coordinate  $\mathbf{x}$ , meaning the polychromatic CT images can be reconstructed. Given a polychromatic X-ray that consists of multiple monochromatic X-rays at  $N$  energy levels, its equivalent monochromatic X-ray's energy can be approximated by  $\bar{E} = \frac{1}{N}\sum_{i = 1}^{N}E_{i}$ . In this work, we aim to solve the CT MAR problem. Thus, we employ the LACs  $\mu_{\bar{E}}$  at the energy level  $\bar{E}$  as our final reconstructed image.
In this section, we aim to answer two key questions: (1) Can our unsupervised Polymer compete with supervised DL MAR techniques for the in-domain and out-of-domain datasets? (2) What is the impact of the key components in our Polymer on the model performance? To study these questions, we conduct comprehensive experiments on three datasets. We provide additional experiment details and results in the supplementary material.
Datasets. We conduct experiments on four datasets, including two simulation datasets and two real collected datasets. The first dataset is DeepLesion [43], the most commonly used dataset for the CT MAR evaluation. For this dataset, we extract 200 2D slices of  $256 \times 256$  size from the original 3D CT volumes as ground truth (GT). Then, we follow the pipeline described in [30, 7, 8] to synthesize metal-corrupted sinograms as input data. The second dataset is the XCOM dataset [44], for which we employ two cases provided by Zhang et al. [7]. The third is a real-world dataset. As shown in Fig. 3 (Left), we insert a metal paper clip into a walnut sample and scan it with a commercial Bruker SKYSCAN 1276 micro-CT scanner. The fourth is also a real-world dataset. We use the same micro-CT scanner to scan a mouse tight containing a metal intramedullary needle. Note that all data solely are utilized for testing purposes since our method is fully unsupervised.