Reliable Evaluation of MRI Motion Correction: Dataset and Insights
Authors: Kun Wang
Tobit Klug
Stefan Ruschke
Jan Kirschke
Reinhard Heckel
Venue: ICLR 2026 Poster (2026)
Keywords: 3D MRI motion correction
Accelerated MRI
Dataset
Evaluation approach
Image reconstruction
Primary area: datasets and benchmarks

Abstract: Correcting motion artifacts in scientific and medical imaging is important, as they significantly impact image quality. 
However, evaluating deep learning-based and classical motion correction methods remains fundamentally difficult due to the lack of accessible ground-truth target data. 
To address this challenge, we study three evaluation approaches: real-world evaluation based on reference scans, simulated motion, and reference-free evaluation, each with its merits and shortcomings. 
To enable evaluation with real-world motion artifacts, we release PMoC3D, a dataset consisting of unprocessed $\textbf{P}$aired $\textbf{Mo}$tion-$\textbf{C}$orrupted $\textbf{3D}$ brain MRI data. 
To advance evaluation quality, we introduce MoMRISim, a  feature-space metric trained for evaluating motion reconstructions. 
We assess each evaluation approach and find real-world evaluation together with MoMRISim, while not perfect, to be most reliable. 
Evaluation based on simulated motion systematically exaggerates algorithm performance, and reference-free evaluation overrates oversmoothed deep learning outputs.

Published as a conference paper at ICLR 2026
RELIABLE EVALUATION OF MRI MOTION CORREC- TION: DATASET AND INSIGHTS
Kun Wang', Tobit Klug", Stefan Ruschkei, Jan S. Kirschke', and Reinhard Heckel***
"School of Computation, Information and Technology, Technical University of Munich Schooo of Medicine and Health, Technical University of Munich
MMunich Center for Machine Learning
ABSTRACT
Correcting motion artifacts in scientific and medical imaging is important, they significantly impact image quality. However, evaluating deep learning- and classical motion correction methods remains fundamentally difficult due the lack of accessible ground-truth target data. To address this challenge, we study three evaluation approaches: real-world evaluation based on reference scans, sim- ulated motion, and reference-free evaluation, each with its merits and shortcom- ings. To enable evaluation with real-world motion artifacts, we release PMoC3D, a dataset consisting of unprocessed Paired Motion- Carrion-rrrupped 3D brain MRI data. To advance evaluation quality, we introduce MoMRISim, a feature-space metric trained for evaluating motion reconstructions. We assess each evaluation approach and find real-world evaluation together with MoMRISim, while not perfect, to be most reliable. Evaluation based on simulated motion systematically exagger- ates algorithm performance, and reference- free evaluation overrates oversmoothed deep learning outputs. Overall, these contributions advance the emerging science of benchmarking for machine learning based scientific and medical imaging, by providing datasets, metrics, and systematic evaluations for motion correction.
INTRODUCTION
Magnetic resonance imaging (MRT) provides rich anatomical detail but requires long acquisition times. Even healthy adults struggle to remain still for several minutes, and small head movements introduce inconsistencies in k-space that lead to blurring, ringing, and spatial misalignment in the reconstructed images. Motion artifacts can prevent effective diagnosis and may force repeating the scan (Andre et al. 2015) Slipsager et al., 2020), increasing costs and reducing the reliability of clinical workflows. Robust motion correction is therefore essential for ensuring dependable MRI acquisition.
Many recent approaches tackle motion correction retrospectively, estimating and correcting motion directly from the acquired k-space or reconstructed images without relying on external motion- tracking hardware. Several works (Haskell et al.) 2019 [Singh et al. 2022 2023 Levac et al.) 2023) proposed methods for reconstructing 2D motion certuin-corrupeed slices. However, subject motion occurs in full 3D, making the problem more challenging and clinically relevant. Prior work has shown promis- ing 3D motion-estimation performance (Johnson & Drangoval 2019; Duffy et al., 2021: Klug et al.) 2024). Cordero-Grande et al.) 2016) introduced classical altemating-optimint optimization framework for jointly estimating motion and reconstructing corrupted volumes. (AI-Masni et al.) 2022) proposed a stacked U-Net that performs end-to-end 3D) reconstruction without explicit motion modeling. More recently, (Wu et al. (025) presented an implicit neural representation method that jointly estimates motion and reconstructs images in both 2D and 3D MRI.
However, research on 3D motion correction is challenging as the field lacks standardized evalu- ation approach for realistically evaluating different approaches. The core issue is that ground-truth target data is fundamentally difficult or impossible to obtain:
Real- world motion cortunt data captures true motion but lacks ground-trun truth for quanti-
tative assessment. To enable real-world evaluation, can collect scans, motion-

---

Published as a conference paper at ICLR 2026
corrupted and motion-free one, and use the motion-frre target or ground-truth. How- ever, the two scans need to be aligned, require careful pre- pre-pescessng and the motion-frie -ree scan might also be slightly motion corrupted.
Most commonly, evaluation is conducted based on simulated rigid motion artifacts in which case computing reference-based metries is straightforward. However, evaluation might be unrealistic due to the motion simulation. Potential artifacts from non-rigid motion are not accounted for when simulating motion (Spieker et al., 2024p. In addition, data has to be fully sampled to simulate motion, which is rarely the case for 3D MRI.
Finally, evaluation can be conducted with reference-free image quality metrics, which avoid the need for rreprrcessinng or the lack of a realistic reference. However, classical gradient- based metrics correlate poorly with perceived image quality (Marchetto et al. 2024).
In this work, we advance evaluation of motion reconstruction algorithms by systematically assess- ing real-world, simulated, and reference-free evaluation, and by providing PMoC3D, a real-world dataset for evaluation of 3D-motion correction methods. Our dataset, PMoC3D, consists of raw Paired Motion- Corruptdd 3D brain MRI data for enabling real-world evaluation by comparing re- constructions motion-corrupted data reference scan. For PMoC3D, collected three motion- corrupted scans of different motion severity each from eight subjects along with one motion-free scan to use as reference.
First, we study real-world evaluation with a reference scan based on the PMoC3D dataset by assess- ing how well reference-basee based image quality metrics correlate with human assessment. We consider standard metrics in the pixel-space such as SSIM (Wang et al.] 2004) and PSNR (Horé & Ziou, 2010) and feature-space metrics such as DreamSim (Fu et al. 2023), and we additionally propose a feature-space metric MoMRISim that is trained to align with varying levels of motion severity. We find that reference-baseee based evaluation using feature-space metries like MoMRISim correlates well with human judgments and provides reliable measure of reconstruction quality. However, under mild motion, the motion-free reference reconstructions often retain residual artifacts, and in some cases, mildly motion-corrupted scans reconstructed with motion-correction methods appear visually cleaner than the reference. This challenges the reliability of reference- based evaluation in mild mo- tion settings, where simulated data with known ground truth can offer more meaningful alternative for evaluation.
Second, assess evaluation based on simulated motion corruption, and observe that methods achieve almost error free reconstructions under the most severe simulated motion, whereas the same methods exhibit noticeable residual artifacts under severe real-world motion. This is consistent with findings for other imaging problems, that found simulated data to potentially lead to misleading conclusions (Shimron al. 2022)
Third, regarding reference-free evaluation, we propose and consider a vision-language model (VLM) score. While exhibiting significantly better alignment with perceived image quality than classical gradient-based reference-free metrics, we find the VLM score to be biased towards recon- structions, which are overly smooth but potentially miss anatomical details.
All three considered evaluation methods have shortcomings, but evaluation on real-world paired datasets such as PMoC3D. when combined with an appropriate feature- based metric such as MoM- RISim, provides a relatively reliable and meaningful assessment of reconstruction performance un- der moderate to severe motion. Overall, these contributions advance the emerging science of bench- marking for machine learning based scientific and medical imaging, by providing datasets, metrics, and systematic evaluations for motion correction.
2 THE PMOC3D DATASET FOR REAL-WORLD EVALUATION
We constructed the PMoC3D dataset in order to evaluate accelerated 3D motion correction methods. PMoC3D is 3D dataset containing the raw measurement data of scans with real-world motion as well as corresponding motion- free scans target estimate of the ground-truth.
Previous works relied on evaluation datasets that provide only the processed magnitude im-
ages (Johnson & Drangova 2019 Duffy et al. 2021 Ganz & Eichhorn 2022 Li et al. 2024
2

---

Published as a conference paper at ICLR 2026
L.(87_3) PMAS: PMAS: 0.36
L.(S5_3)
L1(86_2) PMAS:
L1(88 PMAS: 1.40
L1(82.1)
PMAS:
Figure 1: Sagittal views and corresponding perceived motion artifact scores (PMAS) of selected Ll-reconstructions from our dataset with varying degrees motion artifacts ranging from mild (S7.3, S5 2) to moderate (S6.2) 2) to severe (S8 2, S2. These examples highlight the challenges in reconstructing motion-corrupted scans.
However, approaches that explicitly estimate motion in order correct for it, require access the unprocessed raw data (k-space data) (Cordero-Crrande et al., 2016 Klug et al. 2024).
2.1 ACQUIRING PAIRED MOTION-FREE AND MOTION-CORRUPTED DATA
For the PMoC3D dataset we scanned 8 healthy male and female subjects. The study was approved by the institutional review board and written informed consent obtained from all participants prior to data collection (see Ethical Statement at the end of the paper for details). From each subject acquired four scans, one motion-free and three motion-corrupted labeled as (subjeect) (oottt)((ssaa) {scan} with subjects 1,....& 8 and scans ., where O corresponds to the motion-free case. When we observed artifacts the scanner's reconstruction while recording a motion-free scan, we assumed that the subject has moved and restarted the scan session. We provide access to the corrupted scans (S3.4, 4, S5 4, S8 4) with involuntary motion resulting in a total of 27 motion-corrupted scans.
Data acquisition. The data was acquired on an Ingenia Elition 3.OT X scanner (Philips Healthcare, Best, The Netherlands) with the standard 16-channel dStream HeadSpine coil array (of which 13 channels were automatically selected based on SNR). We performed Cartesian 3D T1-weighted Fast- Mast-Gradient-ccho (TFE) imaging with I mm isotropic resolution and field of view of 221 x 170x 256 mm. With k and ky oversampling of 1.4 and 2.0, the acquisition matrix-size ky x k- x liz 222 x 236 x 512.
Data was acquired with undersampling along the phase encoding dimensions ky x h2 (axial plane) with undersampling factor of μ 4.94, densely sampled auto-calibration region of size 37 37 and partial-Fourie sampling with factor 0.85 the kg direction(see Figure in Appendix B for the resulting undersampling mask).
We provide access to the full-size k-space data. To reduce the computational cost for evaluating experiments, we also cropped the data, where crop along the fully-sampled read-out dimension k2 to the size of the field of view (256) by subsampling every second voxel. The sequence parameters in Appendix B
Sampling trajectory. The Cartesian k-space is acquired within N, 52 shots resulting in ky kz/(NN,) = 222 236/(4.94 52) = 204 acquired read-out lines per shot. The acquisition of one shot lasts 1.35s followed by pause of 1.74s resulting in total scan duration of 2:40min. The read-out lines are acquired following quasi-random sampling trajectory except the 3 x 3 center of the k-space which is acquired the start of the first shot. Hence, the sampling order varies between all scans in the dataset.
We chose random order because it ensures that both low-thigh-frequency components are sampled in every shot which is beneficial for motion estimation (Cordero-Grandd et al. 2016 Usman et al, 2020 Klug et al. 2024).
Sensitivity maps. For each subject, the dataset contains the coil sensitivity maps calculated using Gyrotools MRecon (LLC) through a calibration scan performed at the beginning of each subject's scan session for which the subject instructed hold still.
3

---

Published as a conference paper at ICLR 2026
Table 1: Overview of real-world paired, simulated motion, and reference-free evaluation.
Evaluation Method
Real-world Paired
Simulated Motion
Reference-Free
Data Source
PMoC3D
Calgary Campinas Brain MRI dataset (Souza et al.
PMoC3D
2018
Reference Type
Paired reference- free scan
Perfect ground truth
None
Eval Metrics
PSNR, SSIM, DISTS MoMRISim....
PSNR, SSIM
AES, VLM scores
Human Alignment
High
Low
Limitations
Imperfect reference-fre free
Simulation fails reproduce real motion complexity
Weak correlation with human evaluation
Favors oversmoothed images
Conclusion
Not perfect. but most reli- able
Overestimate algorithm per-
Unreliable
formance
Motion corrupted scans are obtained by instructing the subject to perform motion at one or more time instances during the scan. The instructions are follows:
Motion.
Slightly turn head left/right and stay/return to origin.
Nodding: Once up, once down and return to origin.
Head shaking: Once left, once right and return to origin.
Move chin towards chest and stay/return to origin.
We instruct to perform motion slowly it should be performed slowly opposed abruptly. To generate a diverse dataset containing mild to severe motion artifacts we vary the instructions itself, when give them, and how many we give (up to three). The instructions and time stamps provided with the data.
2.2 CATEGORIZING MOTION ARTIFACTS IN THE DATA
Stronger motion and more motion events make reconstruction more difficult. To facilitate evaluation with the PMoC3D dataset, we provide a quantitative measure of human-perceived severity of the motion artifacts of each of the scans corrupted with voluntary motion.
The score is computed as follows. We first obtain the Ll-reconstruction with wavelet regulariza- tion (Lustig et al. 2008) without any motion correction for each scan (see Appendix [D. for de- tails). The Ll-reconstruction with wavelet regularization does not account for motion modeling or correction, allowing us to directly evaluate the severity of motion artifacts in each scan.
Two PhD students with expertise in machine learning and MRI reconstruction performed pairwise comparisons between the reconstructions, classifying either one have more or both to have sim- ilarly severe artifacts. Since PMAS here is used only to categorize the severity of motion arti- facts in the raw scans, the use of trained non-clinical evaluators is sufficient for this purpose. both evaluators agree that reconstruction A has p(A > B) = If one evaluator judges A to be better and the other finds a similar level, then we assign p(A B) 0.75. If both evaluators find similar severity level, one finds one better and the other the other, we set p(A > B) 0.5.
severe artifacts than then assign rate
Based on these pairwise results we fitted Bradley-Terry model (Bradley & Terry 1952) obtain a perceived motion artifact score for each scan:
exp(3.) Σ j) log exp(33) + exp(83) (
PMAS = arg max
(1)
where each 3g quantitatively represents the severity of motion artifacts for the corresponding vol- ume; higher values indicate more severe artifacts, and these latent parameters serve as our measure of perceived motion artifact severity(see Appendix for more details).
4

---

Published as a conference paper at ICLR 2026
Figure I shows sagittal views of L1 .--reconstruccions from our dataset from left to right in ascend- ing order according to their perceived motion artifact score. As we can see, the severity of motion artifacts increases with increasing perceived motion artifact score. Reconstructions S7 3 and S5 show mild artifacts, where most brain anatomical details are preserved despite the presence of mi- ringing artifacts. In reconstruction S6_2, the artifacts
more pronounced and obscure finer details, while in S8 2 and S2-1 the artifacts are severe enough that the brain structures become barely discernible. These examples illustrate the range of challenges encountered when reconstruct- ing cootion-corruptd scans in our dataset.
3 EVALUATION APPROACHES
Evaluating motion correction in 3D MRI is challenging because we do not have the perfect reference. We have considered 3 evaluation methods, each with its own strengths and failure modes. Table [ provides an overview comparison of the three evaluation methods considered in this work.
In the remainder of this section, we describe these three evaluation approaches in detail: real-works evaluation with a reference scan, evaluation based on simulated motion, and reference-free evalua- tion. We also discuss standard evaluation metrics and propose two novel metrics for the respective approaches.
3.1
EVALUATION WITH A REFERENCE SCAN
We perform real-world evaluation with our PMoC3D dataset, which consists of paired acquisitions for each subject. Each subject Si (i=l....,8) undergoes one motion-free sean Si.0 and three motion- corrupted scans Si.j (j=1.2.3). The motion-corrupted scans categorized into two difficulty levels: the scans with the lowest perceived motion artifact scores (PMAS) are labeled as mild motion- corrupted scans, while the remaining scans classified as moderate and severe. Each baseline method is applied to each motion-corrupted scan, and the LI reconstruction of the motion-free scan the reference for quantitative scoring.
We preprocess the data to mitigate the challenges of comparing two different acquisitions, suggested by the paper (Marchetto et al.] (024). First, using advanced normalization tools(ANTs) (Tustison et al. (021) rigidly aligns the motion-corrupted volume to the reference. Subsequently, brain mask is generated via BET (Smith) 2002) on the motion-free scan and applied to both datasets, to focus the evaluation on the anatomical region of interest. Both volumes are mul- tiplied by the brain mask and normalized to a max value of the 99.9th percentile before computing scores. After preprocessing, reference-basee quality metrics are computed between the corrected motion-corrupted scans and the Ll-based motion-free reference reconstructions (Si.0).
3.2 EVALUATION BASED ON SIMULATED MOTION
We evaluate performance by generating synthetic motion-corrupted, undersampled k-space data from the fully sampled Calgary Campinas Brain MRI dataset (Souza et al.) 2018), and comparing reconstructions against the original, uncorrupted reference volumes. This has the advantage that we have accurate ground- truth or target information, and the disadvantage that the motion is synthetic.
A 3-D Cartesian mask with acceleration factor of 4.9 is applied in the two phase-encoding direc- tions, replicating the mask geometry used for PMoC3D. Each acquisition is divided into 52 shots following random trajectory, again mirroring the paired real-world protocol. Inter-shot head mo- tion generated with an event-based framework designed to resemble PMoC3D artifacts. Motion events follow the instructions in Section 2 (head turning, nodding, etc.) which involve rotations about the ky-kiz and kzzk, axes. To more realistically capture head motion, we introduce random perturbations the remaining motion parameters, i.e., the three translational components and the third rotational axis. These perturbations account for natural subject-specifi variability and for the fact that real-world head rotations often occur around off-center axes rather than the image origin, resulting in complex motion patterns. Motion severity is controlled by the number of events and their amplitude:
Mild: One event with primary motion sampled uniformly from + and perturbations up to /mm.
5

---

Published as a conference paper at ICLR 2026
Severe: Three events with primary motion sampled uniformly from +15 and perturbations up to /mm
To match the behavior of our real acquisitions, the simulation parameters are chosen mirror those in PMoC3D. Mild motion contains single event, and severe motion contains three events, consis- tent with the PMoC3D acquisition protocol. The motion amplitudes were selected based on visual inspection of the real severe 1.1 reconstructions to ensure that the simulated artifacts are at least as challenging as the most corrupted in-vivo scans.
For each baseline, we evaluate performance across ten test volumes. For every volume, we simulate the two severity levels, and for each level, we draw two independent motion seeds. Reconstructed volumes are normalized their 99.9th percentile intensity, after which reference- based metrics are computed against the fully sampled ground-truth images. While simulated motion offers control- lable and repeatable conditions, it may oversimplify the motion dynamics in vivo.
3.3 REFERENCE- FREE EVALUATION
In the reference-free setting, each baseline method reconstructs the motion-corrupted scans Si.j. for in I.....,, and j from to on PMoC3D. The reconstructed volumes are first masked to exclude non-brain regions. consistent with the paired evaluation protocol. Subsequently, the volumes are then normalized by the 99.9th percentile normalization, matching the normalization used in the paired evaluation. Because the available motion-free scans cannot serve as true ground truth, the normalised reconstructions are evaluated directly with a reference-free metric discussed in the next section.
3.4
EVALUATION METRICS
We first review existing reference-based and reference-free metrics, then introduce our two proposed metries: MoMRISim (reference-based) and the VLM score (reference-free).
3.4.1 EXISTING METRICS
Reference-based metrics. We consider standard pixel-wise metries including Structural Similar- ity Index (SSIM) (Wang et al. 2004), Peak Signal-to-Noiie Ratio (PSNR) (Horé & Ziou 2010). and Artifact Power (AP) (Ji al.) 2007). We also consider perceptual metrics, which evaluate image quality based on high-level visual representations rather than direct pixel-wise comparisons. These metrics leverage deep learning-based feature extraction assess structural and perceptual similar- ity, and have demonstrated strong correlation with radiologists' assessments (Adamson et al. 2025)
Specifically, we consider Deep Image Structure and Texture Similarity (DISTS) (Ding et 2022) and DreamSim (Fu et al. 2023).
Reference-free metries. We also investigate reference-free metrics for assessing image quality without requiring access to a motion-free reference scan. We employ two gradient-based methods, Average Edge Strength (AES) (Pannetier et al. 2016) and Tenengrad (TG) (Kecskemeti et al. 2018)
which quantify image sharpness and structural clarity.
3.4.2 MOMRISIM FOR REFERENCE-BASER EVALUATION
Existing image-quality metrics struggle to capture motion artifacts in 3D MRI. Pixel-based met- rics such as PSNR and SSIM exhibit only moderate correlation with human evaluation. Perceptual metrics are trained to model semantic similarity in natural images, rather than the motion artifacts MRI. Moreover, adapting perceptual metries like DreamSim (Fu et al., 2023) typically requires large-scale human annotations, which impractical to collect for subtle 3D motion artifacts. To overcome these limitations, we propose MoMRISim, a perceptual similarity metric tailored specif- ically motion artifacts in 3D MRI. Unlike DreamSim, which is trained on human-labeled natural- image triplets, MoMRISim utilizes motion severity as a self-supervised signal and is therefore opti- mized directly for motion artifact discrimination.
Our goal is to learn an encoder f(-) whose feature space reflects motion severity through its distance to the motion-free reference. In this space, mildly corrupted reconstructions should lie closer their

---

Published as a conference paper at ICLR 2026
reference than severely comrupted ones. To obtain such an encoder, train it using triplets consist- ing of motion--fre reference and two motion-coruptedd reconstructions with known simulated severities. Because the severity ordering in each triplet is exact by construction, the model receives perfect relative supervision without requiring any human annotations. The encoder is optimized so that mildly corrupted images embedded closer to the reference than their more severely corrupted counterparts. Triplets are constructed by applying synthetic rigid motion of varying severity, defined following the protocol in paper (Klug et al. 2024), to the fully sampled Calgary Campinas Brain MRI dataset (Souza et al. 2018). To enhance robustness across reconstruction styles, we randomly apply either L1- or U-Net- based reconstruction without motion correction, encouraging the encoder to learn motion-artifact features that are invariant to the two reconstruction pipelines used in this study.
At evaluation time, the triplet structure is no longer needed: MoMRISim reduces to standard dnference-reconstruee pair. MoMRISim assigns a motion severity score to reconstruction rel- ative to its reference R by computing the cosine distance between their embeddings:
MoMRISim(RR, R, X)) CosineSimilaritty (f (
where f(-) is the learned feature extractor. Higher values indicate greater deviation from the refer- ence and, therefore. more severe motion artifacts. Full training details, including data preparation and hyperparameters, are provided in Appendix with Figure [ illustrating a triplet-based input example used during training.
3.4.3 VLM SCORE FOR REFERENCE- EVALUATION
Classical gradient-based image quality metries have been shown to correlate poorly with human judgments in prior work (Marchetto et al., 2024). Our own experimental results in Appendix G.1
corroborate this finding. highlighting the limitations of these metrics in evaluating motion arti- facts in 3D MRI. To address this, propose reference-free VLM score based on prompting a vision-language model. We evaluate our approach using GPT-4o (OpenAl 2024b a Qwen2.5-
VL-Max (Bai et al. 2025). Med3DVLM (Xin et al., 2025), and M3D-LaMed (Bai al. 2024). The
models are asked to assign motion artifact severity score ranging from 0 (no motion) to (severe motion). To enhance robustness, we prompt the model independently five times a temperature of 0.5 and compute the final score as the average across all runs. See Appendix F for details.
ASSESSING EVALUATION APPROACHES
In this section, we assess the three evaluation approaches.
4.1 IMPLEMENTATION DETAILS AND BASELINES
We evaluate three cootion-reconstructinn methods that span the major paradigms used in 3D MRI motion correction: classical optimization-basee method that relies purely on physics, a hybrid deep-learning and physics approach, and a fully end-to-end deep learning model that corrects mo- tion without explicit motion modeling. Covering these three categories allows to assess metric performance across fundamentally different reconstruction strategies. Following three motion re- construction methods are utilized generate reconstructions for evaluation:
The classical alternating optimization (AltOpt) (Corder--Grande et al. 2016) alternately optimizes the L1-wavelet reconstruction and the motion parameters, updating one while holding the other fixed.
The deep learning-base MotionTTT (Klug al. 2024) relies on 2D U-net (Rnnebergerr et al. (205)) pre-trained to perform motion-free free MRI reconstruction. Motion estimates the 3D motion parameters by optimizing data-consistency (DC) loss between the network output and the given motion-corrupedd measurements over the motion parameters.
E2E Stacked U-nets (Al-Masni et al.) 2022) is based stack of refinement U-nets to predict the motion-corrected reconstruction sisce-wise. The training details are described in Appendix D.4
7

---

Published as a conference paper at ICLR 2026
McMiRiSSa
VIMI Scee
ABS0 MirkoglTT
Stadled Usee
Rank comparison between PSNR, MoMRISim, and VLM score (GPT-40) and the per- ceived motion artifact score (PMAS). Both axes show rank values rather than scores. A smaller rank indicates a better reconstruction. Points lying closer to the diagonal reflect stronger agreement between metric's ordering and the PMAS ordering. MoMRISim shows strong correlation, PSNR offers moderate level, and VLM score(GPT-4o) reflects low degree of alignment.
Figure 2:
MotionTTT and stacked U-nets are both trained on the Calgary Campinas Brain MRI Dataset (Souza et al. 2018) consisting of TI- W--ehihtted 3D motion-free brain scans recorded under similar setting as our PMoC3D dataset. For MotionTTT and AltOpt, we estimate six motion parameters (three ro- tations, three translations) per acquired shot, and perform L.I-minimzzation based on the estimated motion parameters for the final reconstruction. Shots with motion parameters that have a data con- sistency (DC) loss above a certain threshold are excluded from the measurements. See Appendix D for all training details and hyperparameters.
4.2 ASSESSING PAIRED REAL- WORLD EVALUATION
Paired real-world evaluation is challenging because no truly motion-free ground truth exists. In PMoC3D, the reference volume is acquired in separate minimization, and thus is not perfect ground-truth data. Moreover, it might be slightly misaligned the motion-corrupted scan even if the motion-free scan had motion corruption.
and reconstructed via L1-
Assessing evaluation reliability with human judge. Because the motion-free volume is not perfect ground truth, we evaluate reliability by studying whether the ranking produced by different metrics measuring similarity of reconstructions and references based the PMoC3D dataset agree with human perception.
We performed pairwise comparisons of 250 randomly selected baseline reconstruction pairs, with a licensed medical doctor judging whether one image exhibited more severe artifacts or both had similar severity. The resulting preference matrix was fitted with Bradley-Terry model (details in Appendix D.I,, which yields perceived motion artifact scores (PMAS). To verify the reliability of PMAS ratings. we also collected a full set of 2,556 pairwise comparisons from a second rater for inter-rater analysis (which was not used in any subsequent experiments). The resulting PMAS ranks show strong consistency; see Appendix G.5 for details.
We assessed the association between PMAS and each evaluation metric using Spearman's 's rank cor- relation coefficient. Correlations for PSNR and MoMRISim are displayed in Figure 2 results for all metrics appear in the Appendix G.I Figure 2 shows that PSNR and MoMRISim yield rank- ings consistent with the human judgment (i.e., with the perceived motion artifact scores (PMAS)). MoMRISim, in particular, attains the highest correlation, which is 0.92. This strong association demonstrates that our evaluation is a faithful proxy for expert assessment, confirming the reliability of paired real-world evaluation.
Challenges in the paired real-world evaluation in particular for mild motion. Figure shows the motion-fil free reference image and the Mot reconstruction of scan S3.3, which was ac- quired under mild subject motion, along with the difference of the two images. For this example, the MotionTTI reconstruction of the mild motion scan is slightly better at parts than the reference image, which can be seen in the zoomed region: The Motion reconstruction is free of ringing.
8

---

Published as a conference paper at ICLR 2026
Motion Recon.
Referenee
Difference
Figure 3: Comparison of Mo reconstruction, L1-based motion-free reference, and their difference image. The left panel shows the MotionTTT reconstruction of subject S3_3, which is mild motion scan. The center panel displays the LI reconstruction of the corresponding motion-free -ree reference. The right panel presents the difference image between the two reconstructions.
whereas the LI reconstructed motion-free reference contains faint ring artifacts. Under mild motion, the corrected image can therefore look better than the reference, which problematic.
To summarize, although imperfect reference volumes may render paired real-world evaluation em- pirically unreliable in the mild-motion regime, real-world evaluation is dependable under moderate to severe motion: when combined with robust reference-basee metric such as MoMRISim, it yields performance assessments that closely align with human evaluation. Because PMoC3D includes both instructed and naturally involuntary motion, also perform analysis of the involuntary cases. The results, presented in Appendix G.4
4.3 ASSESSING SIMULATED MOTION EVALUATION
Evaluation based on simulated motion is popular for its simplicity and since motion motion-corrupped real- world data required. However, such simulations may not reflect real-world motion artifacts sufficiently well. We find that evaluation based on simulated data overestimates the performance of reconstruction methods and can therefore be misleading.
This is easy to see from Figure 4 which contains severely corrupted real scan from the PMoC3D dataset as well as a simulated motion with a similar level of motion corruption (as seen the reconstructions in the figure). It can be seen that the reconstructions of all considered algorithms are significantly better for simulated data compared to the reconstructions based on the real data. In addition, Figure 12 in Appendix G.2 compares five simulated cases of severe motion with the five most severe cases from real-world scans, showing results both with and without motion cor- rection. The simulated artifacts were designed be as severe in the real-world data. In all instances, reconstructions from the real-world data retain noticeable ringing artifacts. In contrast, the reconstructions of simulated data using motion correction appear consistently clean, with motion artifacts largely eliminated.
more severe than those found
Our comparison confirms that, even comparable levels of artifact severity, reconstructions from simulated motion consistently appear cleaner than those from real-world motion. This discrepancy indicates that evaluation approaches restricted simulation therefore risk systematically overesti- mating algorithmic progress.
4.4 ASSESSING REFERENCE- EVALUATION
Reference-free evaluation can test for the presence of artifacts, but cannot measure accuracy since no reference is available. We find that reference-free metries systematically overestimate the perfor- mance of the deep learning based method.
To evaluate metric reliability, we examine the correlation between reference-free quality scores and PMAS. As illustrated in Figure 2 right panel, the reference-free VLM score(GPT-40) exhibits weak correlation with the human judge. Comprehensive results for all reference free metrics are reported in Appendix G.I, where consistently low correlations with expert evaluation are observed. A notable failure case involves certain stacked U-net reconstructions, where the VLM score as- signs high quality despite PMAS indicating substantial motion artifacts. This discrepancy suggests

---

Published as a conference paper at ICLR 2026
Reference
Mo
AltOpt
E2E
L1
Simulated Recon.:
Error w.r.. refer- ence
PMoC3D Recon.:
Error w.r.. refer- ence
Figure 4: Comparison of baseline reconstructions on scans with artifacts from both simulation and the real-world PMOC3D dataset. The first column displays the motion-free free reference. Subsequent columns show the results from MotionTTT., AltOpt, E2E Stacked U-Net, and I.1 minimization with- out motion correction. The error maps, shown below each reconstruction, visualize the difference relative to the reference.
that reference-Fie metrics do not reliably align with human perception of artifact severity. In Ap- pendix G.3 we provide example where reference-free metric fails reflect the quality of reconstruction. Although the stacked U-net rated favorably by the metric, the corresponding differ- ence image reveals substantial loss of anatomical detail. Reference-free evaluation is prone bias, particularly for end-to-end deep learning models that suppress artifacts through oversmoothing.
CONCLUSION
Reliable evaluation of motion correction algorithms in 3D MRI fundamentally difficult due the absence of good ground truth in real-world data. This paper presents a comprehensive assessment of three evaluation approaches: real-world evaluation with a reference scan, based on simulated motion, and reference-fre free assessment.
To enable weal-world evaluation, we introduced the PMoC3D dataset consisting of paired motion- free and motion-corrupied scans. We find that real-world evaluation is well correlated with human judgment of reconstruction and is thus relatively reliable. However, for very mild motion, baseline reconstruction methods produce better results than the motion-free reference, which potentially compromises the validity reference-basce evaluation the mild-motion regime. Evaluation based simulated motion can be misleading because simulated motion fails to capture the full complex- ity real-world motion and tends to overestimate performance. However, evaluation based on simulated motion still be useful for relative comparisons.
Reference-free evaluation can very biased towards certain reconstructions and is not reliable, as expected.
10

---

Published as a conference paper at ICLR 2026
Ethical statement. The local institutional review board approved the study (study number 2024- 365-S-NP) in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments comparable ethical standards. Prior informed consent was obtained from all individual participants.
Reproducibility Statement. All data, code, and reconstructed volumes associated with this work are publicly available. Acquisition details for the PMoC3D dataset are provided in Section [ and Appendix B the dataset and the reconstructed volumes are hosted at htthtt::/// httes///huggnnnigg co co/datasets/ ml -aalll--la/// PMoC3D Training and inference configurations for the baseline constructions are detailed in Appendix D and the corresponding code to reproduce these baselines is available at httpt:::pps........ppp / q t.ubb.th......... com/ MLI- MRI MotionTTT Our evaluation methods
are described in Section 3.4, Appendix and Appendix F with the evaluation code accessible at https://githus: htw.thuub.oom........ cww. ML ab PMcCPD ree main
This work was supported by the Deutsche Forschungsgemeinschatf (DFG, German Research Foundation) - 456465471, 517586365 and the German Federal Ministry of Edu- cation and Research, and the Bavarian State Ministry for Science and the Arts.
Acknowledgements.
REFERENCES
Philip M. Adamson, Arjun D. Desai, Jeffrey Dominic, Maya Varma, Christian Bluethgen, Jeff P. Wood, Ali B. Syed, Robert D. Boutin, Kathryn J. Stevens. Shreyas Vasanawala, John M. Pauly, Beliz Gunel, and Akshay S. Chaudhari. Using deep feature distances for evaluating the perceptual quality of MR image reconstructions. Magnetic Resonance in Medicine, 94(1):317-330, 2025. doi: https://ooi.org 10000000000000000/400000
URL.
wwwriinnnibbr.rr....... wilyy.rrrr..................e.e. com/
doi /oii/bs// 10. 1 1002 mrmm 30437
Mohammed A. Al-Masni, Scul Lee, Jacuk Yi, Sewook Kim, Sung-Min Gho, Young Hun Choi, and Dong-Hyun Kim. Stacked U-Nets with self-assisted priors towards robust correction of rigid motion artifact in brain MRI. Neurolmage, 259: 1 19411, October 2022.
Jalal B. Andre, Brian W. Bresnahan, Mahmud Mossa-Basha, Michael N. Hoff, C. Patrick Smith, Yoshimi Anzai, and Wendy A. Cohen. Toward Quantifying the Prevalence, Severity, and Cost Associated With Patient Motion During Clinical MR Examinations. Journal of the American College of Radiology, 12(7):689 695, July 2015. ISSN 1546-1444 doi: 10.10166fjjacr20155.0 007. URL httthtt/////oo/ wwt.:/o...rr/.1... 1016 jacr 2015 03. 007
Fan Bai, Yuxin Du, Tiejun Huang, Max Q. H. Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi modill large language models, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL /atttp://arxiv.oom
abs/ abs/2502. 2200 13923
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-45, 1952.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervupe vision transformers. In /EEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
Lucilio Cordero- Cordero-Grande, Rui Pedro A. G. Teixeira, Emer Hughes, Jana Hutter, Anthony N. Price, and Joseph V. Hajnal. Sensitivity Encoding for Aligned Multishot Magnetic Resonance Recon- struction. IEEE Transactions on Computational Imaging, 2(3):266-28) September 2016.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567-2881, 2022. doi: 10. 10.1109900

---

Published as a conference paper at ICLR 2026
Ben A Duffy, Lu Zhao, Farshid Sepehrband, Joyce Min, Danny JJ Wang, Yonggang Shi, Arthur W Toga, and Hosung Kim. Retrospective motion artifact correction of structural MRI images using deep learning improves the quality of cortical surface reconstructions, Neurolmage, 230: April 2021.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 50742- 50768, 2023.
Melanie Ganz and Hannah Eichhorn. Datasets with and without deliberate head movements for eval- uating the performance markerless prospective motion correction and selective reacquisition in general clinical protocol for brain MRI. OpenNeuro, 2022.
Melissa W. Haskell, Stephen F. Cauley, Berkin Bilgic, Julian Hossbach, Daniel N. Splitthoff, Josef Pfeuffer, Kawin Setsompop, and Lawrence L. Wald. Network Accelerated Motion Estimation and Reduction (NAMER): Convolutional neural network guided retrospective motion correction using a separable motion model. Magnetic Resonance in Medicine, 1444):1152-141,,, October 2019.
Alain Horé and Djemel Ziou. Image Quality Metrics: PSNR vs. SSIM. In International Conference on Pattern Recognition(ICPR), pp. 2366-2369, August 2010.
Jim X. Ji, Jong Bum Son, and Swati D. Rane. PULSAR: A Matlab toolbox for parallel magnetic resonance imaging using array coils and multiple channel receivers. Concepis in Magnetic Reso- nance Part B: Magnetic Resonance Engineering, 31B(():24-36, 2007.
Patricia M. Johnson and Maria Drangova. Conditional generative adversarial network for 3D rigid- body motion correction in MRI. Magnetic Resonance in Medicine, 82(3):901-910, September 2019.
Steven Kecskemeti, Alexey Samsonov, Julia Velikina, Aaron Field, Patrick Turski, Howard Row- ley. Janet E Lainhart, and Andrew L Alexander. Robust motion correction strategy for structural MRI in unsedated children demonstrated with diceediimniional radial MPnRAGE. Radiology, 289(2):509-516, 2018.
Tobit Klug, Kun Wang, Stefan Ruschke, and Reinhard Heckel. MotionTTT: 2D Best-Time- Tramn-Ta Motion Estimation for 3D Motion Corrected MRI. In Advances Neural Information Processing Systems (NeurIPS), November 2024.
Gregory R. Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron O'Leary. Py Wavelets: A Python package for wavelet analysis. Journal of Open Source Software, 4(36):1237, April 2019.
Brett Levac, Ajil Jalal, and Jonathan I. Tamir. Accelerated Motion Correction for MRI Using Score- Based Generative Models. In IEEE International Symposium on Biomedical Imaging (ISBI), pp- 1-5, April 2023.
Feng Li, Zijian Zhou, Yu Fang, Jiangdong Cai, and Qian Wang. MoCo-Diff: Adaptive Conditional Prior on Diffusion Network for MRI Motion Correction. In Marius George Linguraru, Qi Dou, Aasa Feragen, Stamatia Giannarou, Ben Glocker, Karim Lekadir, and Julia A. Schnabel (eds.), Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 411-421. Springer
Nature Switzerland, 2024. ISBN 978-3-031-72089-55
GyroTools LLC. ReconFrame. procucts/reconfren URL https: /tw::://www...
wttp:/ggoooolloccooge//////////e products/reconfram Software, accessed 2025-02-26.
www. www.ww.o
Michael Lustig, David I.. Donoho, Juan M. Santos, and John M. Pauly. Compressed Sensing MRI. IEEE Signal Processing Magazine, 25(2):72-82, March 2008
Elisa Marchetto, Hamnah Eichhorn, Daniel Gallichan, Julia A. Schnabel, and Melanie Ganz. Agree- ment of Image Quality Metrics with Radiological Evaluation in the Presence of Motion Artifacts. ArXiv, pp. arXiv:241. 18389v1, December 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAl. GPT-4o system card, 2024a. URL htt.Lttt.//aaa.....////////21
OpenAl. Hello GPT-4, 2024b. httphtttpes///eeaaa..ccoo//////e
Nicolas A Pannetier. Theano Stavrinos, Peter Ng. Michael Herbst, Maxim Zaitsev, Karl Young, Gerald Matson, and Norbert Schuff. Quantitative framework for prospective motion correction evaluation. Magnetic Resonance in Medicine, 75(2):81--3)) 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Medical Image Computing and Computer Assisted Intervention (MIC- CAI), pp. 234-241, 2015.
Efrat Shimron, Jonathan I. Tamir, Ke Wang, and Michael Lustig. Implicit data crimes: Machine
learning bias arising from misuse of public data. Proceedings of the National Academy of Sciences, (19(1))x22112201111 March 2022. doi: (0..007/pnas.211722333 7/www. pnas
URL https:
10
Nalini M. Singh, Juan Eugenio Iglesias, Elfar Adalsteinsson, Adrian V. Dalca, and Polina Golland. Joint Frequency and Image Space Learning for MRI Reconstruction and Analysis. The Journal of Machine Learning for Biomedical Imaging, 2022:018, 88, June 2022.
Nalini M. Singh, Neel Dey, Malte Hoffmann, Bruce Fischl, Elfar Adalsteinsson, Robert Frost, Adrian V. Dalca, and Polina Golland. Data Consistent Deep Rigid MRI Motion Correction. In Medical Imaging with Deep Learning (MIDL), April 2023.
Jakob M. Slipsager, Stefan L. Glimberg, Jes Søgaard, Rasmus R. Paulsen, Helle H. Johannesen, Pemille C. Martens, Alka Seth, Lisbeth Marner, Otto M. Henriksen, Oline Olesen, and Liselotte Højgaard. Quantifying the Financial Savings of Motion Correction Brain MRI: A Model-Base Estimate of the Costs Arising From Patient Head Motion and Potential Savings From Implemen- tation of Motion Correction. Journal of Magnetic Resonance Imaging. 52(3):731-77 September 2020.
Stephen M. Smith. Fast robust automated brain extraction. Human Brain Mapping, 17(3):143-155, November 2002.
Roberto Souza, Oeslle Lucena, Julia Garrafa, David Gobbi, Marina Saluzzi, Simone Appenzeller, Letícia Rittner, Richard Frayne, and Roberto Lotufo. An open, multi-vendor, multi-field-ste multi-field-strennthh brain MR dataset and analysis of publicly available skull stripping methods agreement. Neurolm- age, 170:482-494, April 2018.
Veronika Spieker, Hannah Eichhorn, Kerstin Hammernik, Daniel Rueckert, Christine Preibisch, Dimitrios C. Karampinos, and Julia A. Schnabel. Deep Learning for Retrospective Motion Cor- rection in MRI: A Comprehensive Review. IEEE Transactions on Medical Imaging, 43(2):846- 859, February 2024.
Nicholas J. Tustison, Philip A. Cook, Andrew J. Holbrook, Hans J. Johnson, John Muschelli. Gabriel A. Devenyi, Jeffrey T. Duda, Sandhitsu R. Das, Nicholas C. Cullen, Daniel L. Gillen, Michael A. Yassa, James Stone, James C. Gee, and Brian Avants. The ANTsX ecosystem for quantitative biological and medical imaging. Scientific Reports, 1 11(1):9068, April 2021.
Muhammad Usman, Siddique Latif, Muhammad Asim, Byoung Lee, and Junaid Qadir. Retro- spective Motion Correction in Multishot MRI using Generative Adversarial Network. Scientific Reports, 10(1):4786, March 2020.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: From error visibility structural similarity. IEEE Transactions Image Processing, 13(4):600-612, April 2004.
Moritz Wolter, Felix Blanke, Jochen Garcke, and Charles Tapley Hoyt. Ptwt The PyTorch Wavelet Toolbox. Journal of Machine Learning Research, 25(80): 2024.
13

---

Published as a conference paper at ICLR 2026
Qing Wu, Chenhe Du, Xuanyu Tian, Jingyi Yu, Yuyao Zhang, and Hongjiang Wei. Moner: Motion correction in undersampled radial MRI with unsupervised neural representation. In International Conference on Learning Representations (ICLR), 2025. URL https///ppeneeveww..neee.
Barrumid=Odnnee fYpo)
Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. Med3dvlm: An efficient vision-language model for 3d medical image analysis. IEEE Journal of Biomedical and Health Informatics, 2025.
14

---

Published as a conference paper at ICLR 2026
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We utilized LLMs in the following two approaches during this research:
For Writing Assistance: LLMs were used to refine the grammar and improve the clarity and readability this manuscript.
For Evaluation: core component our methodology, vision-language models (VLMs) were employed to evaluate motion artifacts. This process is detailed in Section 3.4 and Appendix F
B PMoC3D ACQUISITION DETAILS
In Table provide list of all relevant sequence parameters used for the acquisition of our PMoC3D dataset.
Table 2: Sequence parameters of the PMoC3D dataset.
Parameter
Value
Sequence
3D T1-TFE
Sampling
Cartesian
Flip angle (deg)
TR (ms)
6.7
TE (ms)
3.0 (shortest) non-selective invert 1060 ms
TFE prepulse / delay (ms)
Min. TI delay (ms)
707
TFE factor
204
TFE shots
52
TFE dur. shot acq (ms)
1742 / 1347
Shot interval (ms)
3000
TFE prepulse delay (ms)
1060
Under-sampling factor
4.94
Half-scan factor Y /
0.8.
Number of cattoccalbbation lines
37
Profile order
random
Field of view (FH x AP x RI., mm)
256 x 221 x 170
Acquisition matrix
256 221
Fold-over direction
AP
Fold-over suppression
no
Fat shift direction Water- fat shift (pixels)
1.6
Saturation slabs
no
Figure illustrates an example from the sampled dataset, including the image volume, its corre- sponding k-space representation, and the undersampling mask pattern applied along 2 phase encod- ing directions.
C PERCEIVED MOTION ARTIFACT SCORE DETAILS
In order to evaluate the severity of motion artifacts in the L1 reconstructions, we first shuffle the reconstructions and conceal their labels. Then, two PhD students with expertise in machine learning and MRI reconstruction performed pairwise comparisons between the reconstructions of the 24 motion-com coriipte scans. If both evaluators agree that reconstruction A has more severe artifacts than B then we assign rate p(A B) 1. If one evaluator judges A to be better and the other finds a similar level, then we assign p(A B) = 0.75. If both evaluators find a similar severity level, or one finds one better and the other the other, p A > B) 0.5.
15

---

Published as a conference paper at ICLR 2026
30
Kg
b)
C)
Figure 5: Panel a): schematic visualization of the magnitude of a 3D volume; Panel b): the corre- sponding 3D k-space data. Panel c): the undersampling masks with the color coding illustrating an example of random sampling trajectory indicating which lines along the readout dimension ką are sampled within the same out 52 shots.
For the PMAS score described in Section which is used to compare reconstruction quality across different baselines and scans, annotations were performed by licensed medical doctor. The evaluation also follows pairwise comparison protocol. For each pair of reconstructions, if the annotator judges that reconstruction A exhibits more severe artifacts than B, we assign a rate of p(A> > B) 1. If the two reconstructions are considered to have similar artifact severity, the win rate is set to p A B) 0.5.
Based those pairwise comparisons, fit Bradley-Terry model (Bradley & Terry 1952), which assigns a latent parameter to each reconstruction. The difference these parameters indicates which volume is considered to have more severe motion artifacts. We estimated these parameters by max- imizing the likelihood function
cxp(B:)
PMAS arg max
Σ p(i lo log ki
(2)
(3.) + exp(
using gradient descent with the Adam optimizer in PyTorch. In this formulation, each 3: quantita- tively represents the severity motion artifacts for the corresponding volume; higher values indicate more severe artifacts, and these latent parameters serve perceived motion artifact score.
Table presents each cotion-corrupted scan's perceived motion artifact score in the PMoC3D dataset. Based on the perceived motion artifact score, we categorize the scans into three motion severity levels as follows:
Mild Motion: $1 31,2,S7_3, S3.3, S1_3, $4_2, $5,2, $2.3
Moderate and Severe Motion: S3.2, S4.1, S7L2, S5.3, S6.2 $2.2 ,S8_2, S6_1, S4_3, S3 I, $5 S1 58 1, S6 3, S8. S2.1
This classification facilitates a structured analysis of the reconstruction methods' performance across varying degrees of motion artifacts.
D HYPERPARAMETER CONFIGURATIONS AND IMPLEMENTATION DETAILS OF BASELINES
In this Section, we provide further details regarding implementation and hyperparameter configu- rations used for reconstructing the PMoC3D dataset with classical Ll-minimization (Lustig al. 2008), alternating optimization similar to (Cordero-Grande et al. 2016), MotionTTT (Klug et al, 2024 and E2E Stacked U-net (Al-Masni et al.) 2022
Sections 2.2 and
D.1 LI- LI-MINIMIZATION
We perform LI minimization with the mean reean-qquare--rree loss function and wavelet regularization. We the Haar wavelet implementation of order one from the PyWave vette package (Lee al.
16

---

Published as a conference paper at ICLR 2026
Perceived motion artifact score (PMAS) for each motion-corrupted scan in the PMoC3D dataset. Higher scores indicate more severe motion artifacts.
Table 3:
Scan ID
PMAS
Scan ID
PMAS
Sean
PMAS
Scan ID
PMAS
S2 1
2.417
S4 3
1.552
S4
0.808
S7
-0.156
S8 3
2.230
$6 1
1.405
S3 2
0.771
S7
-0.231
S6_3
2.197
S8_2
1,400
S2_3
0.485
S1_2
-0.443
$8_1
2.189
S2.2
1.102
S5.2
0.356
$1_1
1,828
S6_2
1,041
S4_2
0.346
S5_1 1.782
S5_3
0.867
$1.3
-0,008
S3.1
1.673
S7.2
0.813
S3.3
-0.057
2019) with PyTorch support through the PyTorch Wavelet Toolbox (Wolter et al. 2024). We run 40 steps with stochastic gradient descent, learning rate of 10th and regularization weight x 10-8
D.2 ALTERNATING OPTIMIZATION
To perform alternating optimization as described in Section 4.1 we run SGD with a learning rate of 10 and regularization weight À 10 during the reconstruction steps and a learning rate of 5 x 10-2 during the motion estimation step. In both steps the loss is the MSE between predicted and given measurement. The optimization process capped 500 iterations, but it terminates early if the difference between the losses of the first and second reconstruction step is less than 0.02.
After alternating optimization we perform LI Ll-minimization from scratch based on the estimated motion parameters as described in Section D.I
D.3 MOTIONTTT
To conduct ses-time-trainiing motion estimation with MotionTTT (Klug al. 2024) the model provided by the authors, which is pre-trained on the Calgary Campinas Brain MRI Dataset (Souza et al. 2018) for the task of 2D motion-fre reconstruction from undersampled MRI.
As outlined (Klug et al. 2024) the iterative motion estimation can be conducted three phases, where during phase 2, motion states pertaining to shots that exhibit large data consistency (DC) loss, can be split into several distinct motion states to estimate a more fine-grained motion trajec- tory during phase and 3. This improve the reconstruction quality compared terminating MotionTTT after phase as potentially less measurements have to be discarded during the DC loss thresholding before the final reconstruction.
For the PMoC3D dataset we observed no significant difference between the reconstruction quality of splitting corrupted shots during phase terminating the optimization after phase and directly thresholding the corrupted shots from the reconstruction. The number of read-out lines that are saved from being thresholded during phase lies in the range from 1-5% of the total number of lines, which appears to be too little to make a visual difference in the reconstruction.
Hence, for the results discussed here we reduce the computational costs of MotionTTT by running only phase 1, where one motion state (3 rotation and 3 translation parameters) is estimated per acquired shot. Specifically, we run 80 steps with an initial learning rate of 1.0 reduced by a factor of 2 at steps 50, 60 and 70. All other parameters are set as in (Klug et al.) 2024).
For the final reconstruction we run Ll-minimization as described in Section D.I based on the esti- mated motion parameters, where shots with DC loss larger than threshold of 0.70 are excluded from the reconstruction.
D.4 E2E STACKED U-NET
For the E2E Stacked U-net baseline results we adopt the network design from Al-Masnii et al, 2022), where set the number of channels in first layer of both U-nets to 64 resulting in total
17

---

Published as a conference paper at ICLR 2026
of 15,9M network parameters. We used instance norm instead of batch norm in network found it to give more stable results.
We train the model on the slices of 40 volumes from the Calgary Campinas Brain MRI Dataset (Souza et al. 2018). In every training step one 3D volume is loaded from which the fully sampled target volume is computed well as motion-fres free and motion-corrupted undersampled input volumes. Then, 20/10 slices are selected randomly from the in each plane Va Ty Ia X Ts and Ty X r2 together with the corresponding target slices resulting in total of 90 input-target pairs per training step. Thus, with batch size of 10 the network parameters are updated times per training step and 40 9 360 times per epoch. We train the model for 200 epochs with the SSIM loss and the Adam optimizer with learning rate of 6 x 10 which is decayed twice by a factor of 10 at epochs 130 and 170.
volumes
We use twice many slices from the motion-free input to ensure that the network can achieve high quality reconstructions in the absence of motion. We generate the motion-corrupted volumes based on the inter-shot motion simulation model from (Klug al. 2024), where we focus on very mild motion with either one or two motion events. During motion event 1-6 randomly selected motion parameters are set to a value drawn uniformly from either [-1, 1] degrees/mm or [-2, 2] degrees/mm simulating subject movement in between the recording of two shots. We focus the model training on mild motion for more severe artifacts image details are occluded and thus irreversible lost for reconstruction with an end-to-end approach. Nevertheless, we note that the motion correction capability of end-to-end model is specific the type of motion simulated during training and hence more sophisticated motion simulation could benefit the model's performance in the regime of mild motion.
E IMPLEMENTATION OF MOMRISIM
We propose a motion MRI Similarity(MoRRSiim) as learned perceptual metric to quantify motion- artifact severity in 3D MRI. The basic idea of the MoMRISim is described in Section 3.4 We are going to describe the implementation details on this section.
We train 40 brain volumes from the Calgary Campinas (CC-359) dataset. For each training volume we apply rigid-body motion corruption to the k-space under randomly sampled severity levels, where severity definitions-ranginn from mild to severe in terms of number of motion events and rotation/trans/le reconstruct each corrupted k-space without motion correction by both an LI-wavelet reconstruction and a 2D U-Net. This ensures that MoMRISim observes artifact patterns from both classical com- pressed sensing and deep learning pipelines.
perturbation magnitudes, with definitions following (Klug et al., 2024). And
Training samples are triplets (Ref, C1. C2), where Ref denotes the LI-wavelet reconstruction of the original motion-free volume, and C1, C2 are reconstructions of the same slice under two distinct motion severities. For example, if the severity of C2 lower than that of Cz then Ref should be closer to C1. An example of the triplet shown in Figure
In each epoch, we construct triplet by enumerating all pairs of reconstructions at two distinct severity levels among the seven corrupted volumes. And then randomly select one of the three anatomical planes (axial, coronal, sagittal), sample ten slices per pair, normalize each slice by its 99. -h.eeeeeeeeettlee intensity, and discard any background-only slices. This yields approximately 000 triplets per epoch.
We adapted the same training way as the DreamSim(Fu et al.) 2023) to fine-tune the DINO- vitb vitb16(Caron et al. 2021) visual encoder augmented with LoRA adapters by minimizing hinge- ranking loss. The model was optimized with AdamW (learning rate 3.0 x 10 10-5), LoRA rank 4, and batch size of 8, over 40 epochs. Training conducted on NVIDIA RTX A6000 GPUs and completed in approximately 90 minutes with 4 workers. The final model achieved a triplet-ranking accuracy of 0.933 on the training set.
18

---

Published as a conference paper at ICLR 2026
Sevrity
Reference
Severiyy66
Example triplet from the MoMRISim training dataset. The left image is corrupted by motion (severity level 3) and reconstructed using 2D U-Net. The right image is corrupted by more severe motion (level 6) and reconstructed with an LI reconstruction without motion correction. The center image is motion-free reconstruction using an LI reconstruction.
Figure 6:
Figure 7: An example input for VLM evaluation, where three slices from sagittal, coronal, and axial orientations are arranged in 3x3 grid for assessment.
19

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION OF VLM SCORE
Vision-language models (VLMs) have found extensive applications across various domains. In this study, we employ multiple VLMs, each requiring a specific input format. For GPT-4o (OpenAl, 2024b a) and Qwen2.5-VL-ME. ferent views into a 3x3 grid as their input. Figure 7 provides an example of this setup. Since Med3DVLM (Xin et al. 2025) and M3D-LaMed (Bai al. 2024) support 3D input, we resize the volume to the desired size for each model's input.
(Bai al., 2025 arrange three slice images from three dif-
We evaluated motion artifacts by generating five independent responses for each reconstruction us- ing a temperature setting of 0.5. Each response is categorized into one of four predefined levels: No Motion, Mild, Moderate, or Severe, denoted scores 0, 2, and respectively. The final evalu- ation score for each instance is calculated as the average of these five categorizations. The prompt we used is as follows:
**Takk***
Evaluate the severity of motion artifacts in the provided MRI image using a structured and systematic analysis.
### **hval Iuaaluation Criteria for MRI Image**
** Motion Artifact:*x No visible motion artifacts; excel
diagnostic quality, and minor zeconstruction noise is acceptable
**Mild:**
The majority of brain details are clearly visible, with only miner artifacts that not obscure diagnostic structures; minimal diagnostic impact, and minor reconstruction noise is acceptable.
Moderatte:** Noticeable artifacts that partially obscure critical diagnostic regions; artifacts significantly impact diagnostic interpretation.
**Sever::** Brain structures are predominantly obscured by artifacts, with only the general shape discernible; diagnosis is extremely challenging impossible. ### **Output Template**
**Analyze Brain Structure Visibility**
Does the image look very smooth, potentially losing
significant detail? (Important for scoring!).
Are all major brain details visible (gyri, sulci, ventricles)
Do motion artifacts blur or distort critical brain details?
Are there regions where brain details are completely lest? Assoos Artifact Types and Locaticns**
-Check for ringing effects (where, how severe).
-Identify other motion artifacts (streaking, ghosting) and their severity.
**Oversmooth Assessment**
Does the image look very smooth (like a very high-quality image)?
Are there areas with smooth distortions?
If yes, do you think the image has an oversmoothing problem? The primary MRI image shows [<[eralll assessment] ** meticn artifacts. The final precise motion artifact level is: [No Mootioc//MMd//odderrte/eeeee
If the severity level No Motion/Mild: Re-examine the image. Are all details truly clear? If any structures appear compromised, consider increasing the severity level.
### **onclussion+
After rethinking, the primary MRI image shows ** (overall assessment]]* motion artifacts, and the details are **. Siven these facters, the final precise motion arti fact level is: Severity level: [No Motion/Mi http:///i//mmderaaee/eeeeee
20

---

Published as a conference paper at ICLR 2026
Mente tork
AlrOge
Stadled
Spearraa y
SSgnamran p D
Spearmon 035
P3GA5
cauk
PMAS
Figure 8: Rank comparison of PSNR, SSIM, and AP with the perceived motion artifact score. All of them show moderate correlation with human judgment.
G ADDITIONAL EXPERIMENT RESULTS
G.1 CORRELATION ANALYSIS BETWEEN PMAS AND METRICS ON PMoC3D RECONSTRUCTIONS
This section presents the correlation results between the Perceived Motion Artifact Score (PMAS) and various image quality metrics computed on the PMoC3D reconstructions. The correlations are the figures for all mild, moderate, and severe situation.
Figure 8 shows the correlation between PMAS and traditional pixel-wise metrics. While these met- rics generally reflect the expected trend of increasing degradation with higher motion artifacts, their correlation with PMAS is only moderate. This suggests that pixel-wis wise metrics have limited sensi- tivity to perceptual quality differences caused by motion corruption and may not be fully reliable for evaluating motion-degraded 3D MRI.
Figure 9 presents the correlation between PMAS and feature-based metrics. Overall, these met- rics show strong correlation with PMAS. However, DISTS demonstrates notable failure mode when evaluating reconstructions from the stacked U-net, consistently assigning them abnormally low scores. In contrast, both DreamSim and MoMRISim exhibit more stable behavior and higher alignment with PMAS. Among them, MoMRISim achieves the highest correlation with PMAS, indicating its robustness capturing motion artifacts.
Figure 10 and Figure II presents the correlation between PMAS and reference-free metrics, which overall show poor alignment with human judgment. Among the evaluated metrics, the VLM scores evaluated by GPT-4o get the highest correlation. And the VLM score(GPT-44) also demonstrates significantly better alignment with PMAS than TG and AES. These results suggest that while reference-free free metrics are generally less reliable for assessing motion artifacts, the VLM score(GPT- 40) may offer alternative when references are unavailable.
G.2 COMPARISON BETWEEN SIMULATED AND REAL-WORLD EVALUATION UNDER SEVERE MOTION
Figure 12 shows five sets of reconstructions with and without motion correction under severe mo- tion artifacts, comparing PMoC3D and simulated motion cases. The first two columns show reconstructions without motion correction, which reflect the raw severity of motion artifacts. Both real and simulated seans display strong artifacts. Notably, the real-world LI reconstructions still preserve some anatomical details, while the simulated counterparts often obscure brain structures cnirrel-innicating that the simulated artifact severity comparable to or even greater than that of real-world scans.
The last two columns present the corresponding MotionTTT reconstructions. In all cases, real-world data retains noticeable ringing artifacts, with the fifth row showing particularly obvious artifacts. In
21

---

Published as a conference paper at ICLR 2026
Meete nonk
AnOre
MTt
Stadled
Spearac A B8
Npeamran DMI
Spearro p-
mk
PMAS
Figure 9: Rank comparison of DISTS, DreamSim, and MoMRISim with the perceived motion artifact score. All of them show high correlation with human judgment, while the MoMRISim shows the highest correlation.
TG
AFS
VIM Stome (GPT-4))
Monte rank
A31Osc
MoT
Stardee Unit
Speenee
Spransen 321
Speanann 0.44
10
14
PMASS
Figure 10: Rank comparison of TG, AES, and VLM Score(GPT-40) with the perceived motion artifact score. All of them show poor correlation with human judgment.
GP7.40
XDID LAMID
Maded
Figure 11: Rank comparison of VLM scores evaluated by GPT-4o, Qwen- VL-Max, Med3DVLM, and M3D-LaMed with the perceived motion artifact score. All of them show poor correlation with human judgment.
22

---

Published as a conference paper at ICLR 2026
Real(L noMoCoo
Simulate(LI noMoCo)
ReaiiMotionTT77
Simnlate (inniate
Figure 12: Reconstructions from the PMoC3D (real) and simulated datasets under two reconstruc- tion methods. The first two columns show Ll reconstructions without motion correction, where simulated volumes exhibit motion artifacts of similar or greater severity compared to real-world scans. The last columns display MotionTTT reconstructions: while real scans retain visible motion artifacts, simulated volumes are consistently corrected with minor residual artifacts.
contrast, the simulated MotionTTT reconstructions appear consistently clean, with motion artifacts largely eliminated.
Given that the simulated artifacts at least as severe those in real-world scans, the signifi- cantly better reconstruction quality further confirms that simulation- based evaluation can lead to a systematic overestimation of reconstruction performance in practical settings.
23

---

Published as a conference paper at ICLR 2026
Reference
MoL
Al1Opt
Stacked U-net
VLN Scoee AESEIR
VLM Scorch
VLM Soarchi.1
ARS
AES-
TGO
PMoC3D
Recon.:
Error w.r.t. refer-
Figure 13: Baseline reconstructions of scan S4_2, with the difference images and the calculated reference-fing free scores.
G.3 FAILURE EXAMPLE OF REFERENCE-FREE EVALUATION
In this section, we present an additional failure case of reference- free evaluation. Figure 13 shows reconstructions from different baselines along with their corresponding error maps.
As illustrated, the stacked U-Net reconstruction exhibits substantial loss of anatomical detail, as clearly visible in the error image. This result is qualitatively worse than those produced by Mo- tionTTT and AltOpt. However, due to its visually smooth appearance, the stacked U-Net receives comparable scores from TG and AES, despite its degraded quality.
Even the VLM sCore(GPT-4o) while generally better aligned with human judgment fails in this case, assigning a near wear-perfect score to the stacked U-Net reconstruction. This example underscores a key limitation of reference-fre freee metrics: they can be misled by superficially clean outputs that actually lack critical structural fidelity.
G.4 RESULTS OF INVOLUNTARY SCANS
Our dataset also includes a small number involuntary-motion scans. One these exhibits signif- icant anatomical misalignment with its reference scan and is therefore excluded from the analysis. We examine the two involuntary scans in Figure [4] which show their baseline reconstructions and corresponding error maps relative the references. The observed artifacts primarily consist of faint ringing, similar to that seen in voluntary mild-motion cases. After applying reconstruction methods such as MotionTTT, the resulting image quality is comparable to that of the corresponding reference-free reconstruction, and no unexpected failure modes are observed.
We also analyzed the metric scores of the reconstructions from the two involuntary scans. Starting from the 250 expert comparisons used for voluntary-motion volumes, we randomly added 63 addi- tional pairwise comparisons, including one involving the involuntary reconstructions, and refit the Bradley-Terry model as described in Section 2.2 We further computed the MoMRISim values for the same involuntary reconstructions. The resulting sank-rank relationship is shown in Figure [5] The involuntary scans fall within the normal variability of voluntary motion cases and do not ap- pear as outliers. This figure indicates that the available involuntary scans do not exhibit abnormal behavior relative to the voluntary cases.
24

---

Published as a conference paper at ICLR 2026
Reference
Motion
AltOpt
U-net
Lt (no MoCo)
S5
Error w.r.t. refer- ence
$8_4:
Error w.r.t refer- ence
Figure 14: Baseline reconstructions of involuntary motion scans S5 4 and S8 4, with the difference images and the calculated reference-free scores.
MoMRISim
Voluntary
70
Involuntary
Metric rank
10
20
30
40
70
PMAS rank
Figure 15: Rank comparison of MoMRISim with the perceived motion artifact score for both vol- untary motion and involuntary motion scans.
25

---

Published as a conference paper at ICLR 2026
Inter-rater PMAS rank correlation
70
Speaammnn p= 0.94
PMAS rank (PhD rater)
60
40
80
PMAS rank (Expert)
Figure 16: Rank comparison of the perceived motion artifact score from 2 raters.
PSLA8 mak
PACSS meak
PSMAS mzk
Figure 17: Rank comparison between MoMRISim and the perceived motion artifact score within different reconstruction methods.
G.5 INTER
INTER-RATER RATER ANALYSIS
To assess the reliability of using single rater for PMAS, asked second evaluator (a PhD student with extensive experience in motion-corrupted MRI reconstruction) to independently score all 72 reconstructed volumes. To reduce evaluation noise, this rater performed full set of pair- wise comparisons, i.e., 2,556 comparisons covering all reconstruction pairs, and we then fitted Bradley-Terry model obtain PMAS values for the 72 reconstructions. As shown in Figure 17 the resulting ranking exhibits a very strong Spearman correlation of 0.94 with the medical doctor's s PMAS. This high agreement demonstrates that PMAS reproducible and not dependent on single rater's 's subjective preference.
G.6 WITHIN-METHOD CORRELATION ANALYSIS
Because each scan is reconstructed with multiple methods, a single volume contributes multiple points in Figure 2 To verify that the observed correlation is not driven by differences between construction pipelines, we also computed correlations within each method separately. As shown in Figure 17 using the 24 scans reconstructed by each method, the correlations between MoMRISim and PMAS are 0.87 for AltOpt, 0.81 for Motion and 0.77 for the Stacked U-Net. The consis- tently high within-method correlations indicate that MoMRISim reflects motion severity rather than method-specifie variation.
26


Reviews:
[{"review_id": "wOuiPNKGqG", "reviewer": "Reviewer_PYGN", "summary": "The manuscript investigates how to **reliably evaluate 3D MRI motion correction** algorithms when ground truth is unavailable. It introduces **PMoC3D**, a paired real and motion-corrupted brain MRI dataset with raw k-space data, and **MoMRISim**, a learned feature-space similarity metric trained through triplets representing different motion severities. The authors comprehensively compare real-world, simulated, and reference-free evaluation paradigms. The results are convincing: on PMoC3D, the proposed feature-based metric (MoMRISim) correlates best with expert judgments, while simulation-based evaluation tends to overestimate performance, and reference-free metrics (including a VLM score) may favor oversmoothed outputs. The paper provides thorough details about data acquisition, PMAS (Perceived Motion Artifact Score) construction, baselines, and implementation.", "questions": "1. In line 36, the paper mentions the work of **Wu et al. (2025)**. As far as I know, that work also conducts **3D MRI motion correction** experiments and adopts a fully 3D representation rather than stacked 2D slices. Given its strong relevance, the authors should discuss this work in the section reviewing related 3D motion-correction studies.  \n2. In Figure 1, the caption appears to contain a small error. The labels “S7_1” and “S5_1” do not match the figure annotations.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- **Well-scoped and clearly structured problem framing.**  \nThe paper provides a well-defined problem setup and a thoughtful taxonomy covering paired, simulated, and reference-free evaluations. It clearly explains why obtaining ground truth is inherently difficult in 3D MRI, which makes the motivation convincing and the study design logical.\n\n- **PMoC3D: a valuable paired 3D dataset with raw k-space.**  \nThe authors present PMoC3D, a paired dataset that includes raw multi-coil k-space data, coil sensitivity maps, sampling trajectories, and timestamped motion instructions. The inclusion of raw acquisition data makes this dataset especially valuable to the community.\n\n- **MoMRISim: a motion-aware learned metric.**  \nThe proposed MoMRISim metric is technically sound and well-motivated. It is trained using a triplet-learning strategy on simulated motion severities and simplifies to a cosine distance during testing, making it both practical and interpretable. It serves as an MRI-specific perceptual proxy with clear relevance.\n\n- **Careful and realistic experimental design.**  \nI find it particularly commendable that the authors conducted every simulation and validation with great attention to detail, making the setup as realistic as possible. This level of experimental care is not common and significantly increases the credibility of the findings.\n\n- **Meaningful impact and insights for the community.**  \nOverall, this work has substantial value for the MRI motion-correction field. The authors convincingly show that existing approaches perform far from ideal in real motion correction and, through diverse experiments, provide numerous insights into evaluation strategies and failure modes. The study feels genuinely impactful, and I look forward to future developments in this direction.", "weaknesses": "**Limited dataset size and missing motion trajectory data.**  \nThe dataset PMoC3D is an important contribution, but there are some limitations. First, the sample size is quite small, containing only eight subjects. Second, although the dataset includes rich metadata such as raw k-space, motion instructions, and timestamps, it lacks **true motion trajectories**, meaning the exact six-degree-of-freedom head poses over time. Including this information would greatly enhance the dataset’s scientific value and utility. I understand that capturing such detailed motion trajectories is technically challenging, but it could be a meaningful improvement for future versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwFBguBam0", "reviewer": "Reviewer_ygbg", "summary": "This work aims to construct a reliable and standardized evaluation pipeline for the MRI motion correction (MoCo) problem. The main contributions include three parts: 1) Collecting PMOC3D, a real-world MRI dataset; 2) Introducing MoMRISim, a feature-based MoCo metric; 3) Benchmarking several MoCo methods (including both traditional iterative and deep learning-based approaches) on simulated and real PMOC3D datasets. Overall, this work builds a comprehensive and reliable evaluation framework for the MRI MoCo task. It is interesting work.", "questions": "See Weaknesses, please.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The collected PMOC3D dataset includes paired motion-free and motion-corrupted 3D k-space raw data, covering diverse motion types. Its construction and release can enable the development of more advanced deep learning–based MoCo methods, which is a critical contribution.\n- The work explores many quantitative evaluation metrics for MoCo reconstructions, including reference-based metrics (e.g., pixel-level SSIM and PSNR, feature-level DreamSim) and reference-free metrics (e.g., AES, TG, and VIM scores based on LLMs). Moreover, a new feature-based metric, MoMRISim, is introduced. These studies on evaluation metrics may provide valuable tools for assessing MRI MoCo accuracy.\n- Benchmarking several recent MoCo models on both simulated and real datasets provides useful insights that could guide future development of more advanced MoCo approaches.", "weaknesses": "- The PMOC3D dataset only includes 8 subjects. While I understand that MRI data collection is costly and time-consuming, this number is still relatively small. Therefore, the dataset may be more suitable as a test set rather than for training deep learning models.\n- To my knowledge, the combination of deep learning models and physics-based iterative reconstruction is currently a mainstream paradigm for MRI MoCo [1][2][3]. These methods typically involve motion trajectory estimation. However, the proposed dataset does not provide the motion trajectories of the scanned subjects, which may limit its usability for evaluating many advanced MoCo methods.\n- For building PMAS, two Ph.D. students performed the classification of motion types. I believe involving clinical experts in this process would improve the reliability of the annotations.\n- Regarding the proposed MoMRISim metric, the paper lacks a sufficient intuitive explanation. As a reader unfamiliar with DreamSim, I found it difficult to understand the principle and motivation behind MoMRISim (Lines 270–286).\n\n[1] Levac, Brett, et al. \"Accelerated motion correction with deep generative diffusion models.\" Magnetic Resonance in Medicine 92.2 (2024): 853-868.\n\n[2] Wu, Qing, et al. \"Moner: Motion Correction in Undersampled Radial MRI with Unsupervised Neural Representation.\" The Thirteenth International Conference on Learning Representations.\n\n[3] Singh, Nalini M., et al. \"Data consistent deep rigid MRI motion correction.\" Medical imaging with deep learning. PMLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvAnREsZE0", "reviewer": "Reviewer_PvMP", "summary": "This paper presents two main contributions. The first is a curated dataset comprising T1-weighted brain MR images from eight subjects. For each subject, scans were acquired with four distinct levels of induced motion artefacts, including a reference scan presumed to be motion-free. Each image is accompanied by a \"perceived motion artefact score.\"\n\nThe second contribution is the proposal of two novel metrics for quantifying motion artefact severity. The first, MoMRISim, is a reference-based metric. It is an adaptation of the DreamSim approach (Fu et al., 2023) that is specifically trained on data with simulated motion severity levels instead of manually annotated images. The second is a reference-free metric, termed the VLM score, which is derived by prompting a Vision-Language Model.\n\nThe authors conduct an analysis to study and compare the behaviour of these proposed metrics against existing methods using their newly introduced dataset. They investigate the correlation between visual ratings and both reference-based and reference-free metrics, as well as the variations in performance when using synthetic motion-corrupted MRIs compared to images with real motion.", "questions": "- **Figure 2 interpretation**: Could you clarify what each data point in Figure 2 represents? Specifically, does a single subject/image appear multiple times for different reconstruction types, and if so, how does this affect the correlation analysis?\n- **Metric details**: For the VLM score, please specify the exact prompt and the Vision-Language Model used. For MoMRISim, what are the key differences in the training protocol compared to the original DreamSim?\n- **Baseline selection**: What was the rationale for choosing the three specific motion reconstruction methods used in the evaluation?\n- **Motion simulation parameters** (Section 3.2): How were the number of events and the amplitude of the motion simulation parameters chosen? \n\nTypos : \n- In legend for Fig.1, images S7_1 and S5_1 are mentioned, albeit not actually appearing in the figure. \n- Page 7 line ~34: mention of ‘AltOpt’, but abbreviation not previously introduced in first element of bullet-point list in part 4.1\n- Page 9 line ~482: missing d letter “the full complexity of real-world motion and tends to […]”", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- **Valuable dataset**: The creation of a dataset with real MRI scans acquired at different, controlled levels of motion artefact, including a motion-free reference for each subject, is a significant strength. Such data is difficult and costly to obtain, and it provides a valuable resource for developing and rigorously evaluating motion detection and correction algorithms.\n- **Novel and relevant metrics**: The introduction of two new metrics—a tailored reference-based metric (MoMRISim) and a novel reference-free metric using a Vision-Language Model (VLM score)—addresses an important need in the field. The goal of developing metrics that align with human visual perception of artefact severity is highly relevant and practical for automated quality control.", "weaknesses": "- **Poor clarity and structure**: The paper's organisation significantly hinders comprehension. The introduction lacks a foundational explanation of motion artefacts, their impact, and existing detection/correction methods. Key details, such as the derivation of the \"perceived motion artefact score\" (Section 2.2), are insufficiently explained in the main text. Section 3, which covers evaluation approaches, confusingly mixes descriptions of existing and proposed solutions without clear distinction. The rationale for selecting the three specific motion reconstruction methods used for evaluation is not provided, making the experimental design appear arbitrary.\n- **Unclear methodological contributions**: The technical description of the two proposed metrics is unclear. For the reference-based MoMRISim metric, it is not adequately explained how the adaptation of DreamSim differs from the original work beyond the use of simulated motion data for training. For the reference-free VLM score, the prompting strategy and the specific Vision-Language Model used are not described with sufficient detail to understand or reproduce the method.\n- **Ambiguous evaluation**: The design and interpretation of the results are problematic. Figures like Figure 2 are unclear; it is not evident if each data point represents a unique image or if the same subject/image is represented multiple times for different reconstruction types. This raises a critical concern that the observed correlations may be driven by the differences between reconstruction methods rather than by the metrics' ability to assess motion severity within a consistent context. This potential confound undermines the validity of the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WJ4B7MFfG", "reviewer": "Reviewer_YT8x", "summary": "This paper attempts to address the evaluation problem for MRI motion correction algorithms. It contributes a small-scale (n=8) paired dataset (PMOC3D) and a new metric (MoMRISim). The authors use this to argue that simulation-based evaluation is misleading and reference-free metrics are biased.", "questions": "1. Please discuss the core contradiction (Weakness #2): Why should a metric trained on \"unreliable\" simulated data be trusted to evaluate real data?\n2. Please provide an analysis to justify why instructed motion (used in the paper) is a valid proxy for involuntary motion.\n3. Please comment on how the low statistical power (n=8 subjects, 1 rater) affects the confidence in the claim that MoMRISim is the best metric.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper focuses on an important problem of MRI motion correction about reliable evaluation. It provides some experimental evidence pointing out the pitfalls of simulation and reference-free evaluations. The MoMRISim metric shows a high correlation with a single rater on this specific dataset.", "weaknesses": "1. The core contribution (a dataset and a metric) is weak. First, a dataset of n=8 subjects is too small to serve as a meaningful or generalizable benchmark. Second, the concept of paired scans is not novel in the medical imaging domain (e.g., in infant datasets like BCP). The work feels more like a domain-specific dataset paper than a methodological contribution suitable for ICLR.\n2. The paper's entire argument is self-contradictory. The authors' central thesis is that simulated data is \"unreliable\". Yet, they use this \"unreliable\" simulated data to train their proposed metric MoMRISim.\n3. The benchmark's clinical relevance is questionable. All quantitative conclusions are derived exclusively from instructed voluntary motion. The paper fails to provide any analysis of the involuntary motion samples mentioned in the dataset, thus lacking evidence that instructed motion is a valid proxy for the true clinical challenge.\n4. The paper's core claims suffer from low statistical power. The conclusions are drawn from a small sample size of only eight subjects. Critically, the gold-standard PMAS used to validate the MoMRISim metric was established by a single licensed medical doctor, with no inter-rater reliability analysis. This design means the headline claim (MoMRISim's $\\rho=0.92$ correlation) lacks statistical significance. The method's applicability is also severely limited, as the authors concede their reference-based approach is invalid in the mild-motion regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)