Task-Oriented Quantitative Performance Assessments for Comparing PET/CT Systems (2016)

255 Objectives Quantitative assessment of PET/CT scanner performance is typically performed using the NEMA NU-2 standard. However, this standard does not use metrics that directly measure clinical task performance, such as lesion detection, localization, characterization or estimation. In addition, the generalized SNR characterization in the NU-2 standard is based, in part, on the FWHM of a point source in air, which is not relevant for modern scanners that only support non-linear iterative image reconstruction methods. We have developed quantitative performance measurements that relate to lesion detection and estimation, as well as a spatial resolution measurement that is more appropriate for iterative image reconstruction algorithms. Methods Detection: We have designed a phantom that contains multiple marginally-detectable hot spots in a background with a contrast ratio of 2.8:1. Importantly, the phantom background is heterogeneous, i.e. 9lumpy9. Detectability is estimated using a channelized Hotelling observer and quantified by the receiver-operator-characteristic metric of the area under the curve (AUC). Estimation: We used the NU-2 IQ phantom, but activated with 68Ge/68Ga in epoxy, to estimate ensemble variance across multiple scans. Generalized SNR: We added a multi-segment line source to the background region of the NEMA-IQ phantom, allowing simultaneous measurement of spatial resolution and background noise. The FWHM versus noise for several reconstruction methods and settings was evaluated. The proposed measurements were evaluated using two time-of-flight (TOF) PET scanners. The scanners used similar crystal sizes, but different timing resolutions (550 vs. 380ps) and axial length (15 vs. 25cm). The longer scanner utilized silicon photomultiplier technology (SiPM). This SiPM system had a NEMA sensitivity that was approximately 3-fold higher than the reference system. For detectability and generalized SNR, two scans were acquired on both the SiPM and reference systems. Results Detection: Detectability was higher in the SiPM system (average AUC increase of 9.6%). For both systems, the AUC was increased by applying detector response compensation (1.1%), and by the use of timing (TOF) information (6.1%). For comparison, halving the number of counts in the images decreased the AUC by 9.4%. Estimation: When image roughness noise was matched, the variability of the recovery coefficients for the spherical test objects was reduced by 33% for the SiPM system, averaged over a range of reconstruction parameters. Generalized SNR: Expected tradeoffs in line-source FHWM vs. noise were observed, showing that constrained resolution measurements are feasible. For acquisitions with matched activity, the SiPM system generated images at a matched spatial resolution that had NEMA noise measures on average 48±16% lower than the reference system. Conclusions These tests provide a clinically relevant methodology for comparing performance of current and next generation PET systems. As expected, the SiPM system with improved sensitivity and TOF timing resolution, but roughly equivalent crystal sizes, showed improved task performance: Approximately 10% detectability improvement, 33% reduction in estimation variability, and a 50% noise reduction for matched resolution. These tests and comparisons could be used to augment the NEMA NU-2 standard for the performance assessment of modern PET/CT systems.