Title: Reliable Evaluation of MRI Motion Correction: Dataset and Insights
Primary area: datasets and benchmarks
Keywords: 3D MRI motion correction, Accelerated MRI, Dataset, Evaluation approach, Image reconstruction

Abstract: Correcting motion artifacts in scientific and medical imaging is important, as they significantly impact image quality. 
However, evaluating deep learning-based and classical motion correction methods remains fundamentally difficult due to the lack of accessible ground-truth target data. 
To address this challenge, we study three evaluation approaches: real-world evaluation based on reference scans, simulated motion, and reference-free evaluation, each with its merits and shortcomings. 
To enable evaluation with real-world motion artifacts, we release PMoC3D, a dataset consisting of unprocessed $\textbf{P}$aired $\textbf{Mo}$tion-$\textbf{C}$orrupted $\textbf{3D}$ brain MRI data. 
To advance evaluation quality, we introduce MoMRISim, a  feature-space metric trained for evaluating motion reconstructions. 
We assess each evaluation approach and find real-world evaluation together with MoMRISim, while not perfect, to be most reliable. 
Evaluation based on simulated motion systematically exaggerates algorithm performance, and reference-free evaluation overrates oversmoothed deep learning outputs.