Concurrent 3D super resolution on intensity and segmentation maps improves detection of structural effects in neurodegenerative disease
Authors: Brian Avants
Nicholas Tustison
Corey McMillan
Taylor R Gosselin
Roger Gunn
Jacob Yost Hesterman
Venue: NeurIPS 2022 Submitted (2022)
Keywords: brain
perceptual super resolution
MRI
neurodegenerative disease

Abstract: We propose a new perceptual super resolution (PSR) method for 3D neuroimaging and evaluate its performance in detecting brain changes due to neurodegenerative disease. The method, concurrent super resolution and segmentation (CSRS), is trained on volumetric brain data to consistently upsample both an image intensity channel and associated segmentation labels. The simultaneous nature of the method improves not only the resolution of the images but also the resolution of associated segmentations thereby making the approach directly applicable to existing labeled datasets. One challenge to real world evaluation of SR methods such as CSRS is the lack of high resolution ground truth in the target application data: clinical neuroimages. We therefore evaluate CSRS effectiveness in an adjacent, clinically relevant signal detection problem: quantifying cross-sectional and longitudinal change across a set of phenotypically heterogeneous but related disorders that exhibit known and differentiable patterns of brain atrophy. We contrast several 3D PSR loss functions in this paradigm and show that CSRS consistently increases the ability to detect regional atrophy both longitudinally and cross-sectionally in each of five related diseases.


Concurrent 3D super resolution on intensity and segmentation maps improves detection of structural effects in neurodegenerative disease
Anonymous Author(s)
Affiliation
Address
email
Abstract
We propose new perceptual super resolution (PSR) method for 3D neuroimaging and evaluate performance in detecting brain changes due to neurodegenerative disease. The method, concurrent super resolution and segmentation (CSRS), is trained on volumetric brain data to consistently upsample both an image intensity channel and associated segmentation labels. The simultancous nature of the method improves not only the resolution of the images but also the resolution of associated segmentations thereby making the approach directly applicable to existing labeled datasets. One challenge to real world evaluation of SR methods such as CSRS is the lack of high resolution ground truth in the target application data: clinical neuroimages. We therefore evaluate CSRS effectiveness in an adjacent, clinically relevant signal detection problem: quantifying cross-sectional and longitudinal change across set of phenotypically heterogeneous but related disorders that exhibit known and differentiable patterns of brain atrophy. We contrast several 3D PSR loss functions in this paradigm and show that CSRS consistently increases the ability detect regional atrophy both longitudinally and cross-sectionally in each of five related diseases.
10
12
14
15
16
Introduction
18 Magnetic resonance image (MRI) datasets capturing in vivo longitudinal change in the human brain
19 are currently available at unprecedented scale. These data allow us to quantify the complex etiology
20 of neurodegenerative disease during life. fundamental problem in quantifying brain disorders
21 from imaging is that many anatomical structures are small in comparison to image resolution. This
22 caused by not only limited image resolution but also the potentially convoluted shape of the targeted anatomy [1]. Thinner, more oblate and/or curved structures undergo more distortion due
23
24 to sampling-related aliasing in comparison to larger, more spherical structures. These distortions can limit detection power in the context of either clinical trials and/or at the level of patient specific medicine [2, 3]. These results also show that, based on first principles, many disease relevant anatomical structures in the brain, in particular cortical regions, mid- regions and hippocampal subfields, should be quantified at higher resolutions (c.g. ze 0.5mm" or smaller rather than the more commonly available s Imm3). The need for increased resolution is only heightened when considering aging and neurodegeneration where some brain structures may lose half or more their pre-disease onset volume or thickness.
26
28
30
31
32 Perceptual super resolution (PSR) for 2D RGB imagery consistently demonstrates the ability to estimate more "realistic" looking upsampled data in comparison traditional linear nearest neighbor interpolants [4]. While many competitive methods are available, the deep back projection network (DBPN) [5] performed consistently in several competitions including NTIRE 2018 and 2019
34
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

36
[6], AIM 2019 [7] and PIRM 2018 [8]). These large challenges compared dozens of methods with respect to a variety of both pereeptual and reconstruction metrics at different levels of upsampling and noise.
37
38
39
Can the 2D RGB performance advantages of methods like the DBPN translate improvements in the 3D quantification of brain regions as seen in MRI? If so, then PSR for 3D neuroimaging promises to improve quantification by better resolving the brain's internal structures and tissue boundaries. While traditional evaluations of PSR focus on reconstruction error and perceptual impression, these measurements do not provide clinically relevant evidence of PSR's 's value in quantification. One barrier to evaluating PSR's impact on clinically relevant outcomes (segmentation volumes) is that ground truth segmentations do not exist at the super-resolved scale. To address this concern. [9] simulated low resolution magnetic resonance images (MRI) of the brain from high-resolution (HR) images obtained from the Human Connectome Project [10, 11]. They then applied a very deep super resolution (VDSR) model to the simulated data and the high-resolution data and compared the accuracy of an automated cortical segmentation method. This careful evaluation study demonstrated that cortical segmentation on the VDSR images closely approximated the HR data. However, relatively few details are provided about the training of this model and associated loss functions. Furthermore, remains unclear whether these improvements in reconstruction error would translate to the detection of population-level effects in real world data particularly in the aging populations that are the target of the majority of interventional trials for the brain.
40
41
42
43
44
45
48
47
48
49
51
53
54
55
more recent effort in volumetric PSR for medical images [12] proposed SOUP-GAN: Super- resolution Optimized Using Perceptual-tuned Generative Adversarial Network (GAN). SOUP-GAN adopts transfer learning from 2D VGG19 to 3D as proposed in [13] to produce a pseudo-volumetric perceptual metric [14]. Shan et al. used this metric to denoise low-dose computed tomography (CT) images and showed its effectiveness at preserving small anatomical structures. Similarly, the SOUP-GAN effort demonstrates that the pseudo-3D perceptual metric improves both PSNR and SSIM as well as shows visually appealing upsampling for a variety of medical imaging modalities. That is, the surprising utility (in 2D) of VGG weights as a feature space [15] appears to at least partially transfer to PSR in 3D medical imaging.
56
57
58
59
61
63
The current research provides perhaps the first broadly scoped, real world evaluation of MRI PSR for quantification of neurodegenerative disease. Moreover, we demonstrate that a regression network (ResNet) that predicts Tlw image quality can yield directly useful perceptual feature space that performs competitively with pseudo-3D VGG19 features. We build these contributions upon the backbone of set of methods that we call concurrent super resolution and segmentation (CSRS) that extends the proven 2D DBPN to 3D and also includes extra output channel(s) enabling segmentation maps to be upsampled concurrently. We use this framework to test the impact of different loss 71 functions on a set of -ppainnspecific, clinically relevant segmentation measurements related to brain atrophy. Specifically, we evaluate CSRS the quantification of frontotemporal disorders [3] from publicly available longitudinal T1-weighted (TIw) neuroimaging (i.c. MRI). Of the several combinations of losses that evaluate, the best model improves not only segmentation performance (when ground truth is available) but also detection power across all our related disorders: structural changes in behavioral variant frontotemporal dementia (bvFTD), semantic variant primary progressive aphasia (svPPA), nonfluent/agrammatic PPA (naPPA), progressive supranuelear palsy (PSP) and corticobasal syndrome (CBS) each of which impacts known networks in the brain. CSRS with a new perceptual loss based on a shallow ResNet layer performs as well or better than VGG-based models in this test of the practical usefulness of PSR.
64
65
66
67
69
73
74
75
77
78
79
80
81
The primary contributions of this work include:
82
new PSR that upsamples multi-label segmentations at the same time as intensity;
88
a new real world evaluation paradigm for PSR in neuroimaging;
84
comparison of three perceptual loss functions for PSR, two of which are new;
85
demonstration that loss choice impacts detection power in natural history studies of neu- rodegenerative disease. Standard intensity similarity and segmentation overlap metrics, on the other hand, do not discriminate performance between the candidate CSRS options.
86
87
88
Model weights, sample data, and training code will be made publicly available after anonymous review.
89

---

2 Methods
90
91
Software platform: We employ the ANTsX platform [16] version 2.3.5 for anatomical labeling. data augmentation/sampling during model training and to form the tabular data for the statistical evaluation. All MRI processing details follow [16]. Tensorflow 2.6.2 is used for deep learning including ResNet implementation and the CSRS architecture. R version 4.1 is used for statistical analysis with packages lmer and ggplot2. All MRI processing was done on Amazon Web Services parallel cluster with 24 cores and 32GB RAM per process (Intel(R) Xeon(R) Platinum 8259CL
92
93
94
95
96
97
CPU @ 2.50GHz).
98
Data: Human Connectome Project (HCP): We downloaded 1,113 high-resolution 0.7 0.7mm² T1- weighted images from the HCP on which to train CSRS. These Tlw data were acquired using a wwwpetiizatioopppppare
99
100
rapid gradient-echo (MPRAGE) sequence on a customized 3T Siemens Skyra; see [10] for all details of acquisition. As such, these images provide both high resolution and high quality in comparison to the majority of publicly available Tlw MRI. Critically, they provide superior resolution for the thin convoluted cortical layer that critical to the measurement of brain atrophy in frontotemporal disorders. We transformed these data into numpy blocks with randomly selected high- high-resolution 64 patches and paired low-resolution 32 patches. For each patch pair, we also provide a high-resolution binary segmentation and low-resolution downsampled version of that binary segmentation. Each patch segmentation was gained by 2-class k-means performed on the patch where the center voxel's label determines which class (1 or 2) is used foreground. This collection of 16,640 patches is then divided randomly into train (n=16,384) and test sets.
101
102
103
105
107
109
110
Data: Parkinson's 's Progression Markers Initiative (PPMI): PPMI is a longitudinal multi-center clinical study of PD patients and age-matched healthy controls http:/ //www/ . ppm.pppmiiinne PPMI employed(s) over 20 data collection sites with scanners that span the primary manufacturers (Siemens, GE, Phillips), a variety of head coils and also magnet strengths (1.5T, 3T). This heterogene- ity of data collection provides a rich set of w images with highly variable image contrast, resolution and quality. We manually reviewed and labelled 1,431 raw Tlw from PPMI to capture the range of quality in an ordinal scale. This resulted in ground truth dataset with 456 images given grade "A" (superior). 568 given grade "B", 350 given grade "C" and 57 given grade "F" which represents images that are of little to use for quantitative studies of brain structure. We then employed standard 3D ResNet (antspynet. create greate_resnet_____rreeeeeeerreee satpnetttmoee_______ 3d with parameters lowest_rest cessstrresluut 3n-32, number_ of illassffiaaiinn labeeiiaeiaioo__aal_la_ad___e labels=4, cardinality=f 39,424 49,4424004 parameters, 53 3D convo- lutional layers) to learn to predict this scale automatically and reliably from the input Tlw. We denote this network as TIw Quality Rating Resnet (ee)QR Details of training T1 WeRResNet are in Supplementary Information.
111
112
115
115
117
118
119
120
121
122
123
Data: Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & 4-Repeat Tauopathy Neuroimaging Initiative (4RTNI): These inter-related multi-site studies share the goal of improving the quantification of frontotemporal spectrum disorders with both imaging and clinical scores. Like PPMI and HCP, these studies provide longitudinal Tlw images that enable measurement of not only the baseline brain structure differences between controls (individuals without disease Le. normal aging) and disease groups but also differences in rates of change due to neurodegeneration.
125
127
128
129
130
We downloaded and curated 4RTNI and NIFD Tlw data and merged these images into common database. These images were collected at three different sites using protocols consistent with ADNI 3T guidelines [2]. The images overall have median spacing that is isotropically 1m with minority of subjects with out-of-plane spacing up to 1.2mm. As such, these data suit the goals of testing PSR for benefits to the quantification of neurodegenerative disease. After filtering data for very low quality images and the presence of longitudinal data collected within 2 years of bascline, we obtained 128 baseline/171 followup images for controls, 60/1 12 for bvFTD, 38/72 for naPPA, 37/71 for svPPA, 55/70 for CBS and 75/102 for PSP. Further cohort details (age, education, sex, etc) are available in supplementary information. We processed all images consistently and automatically with default
131
132
133
134
135
136
137
138
139
ANTsX pipelines to gain cortical, medial temporal lobe and deep brain structure segmentations for
140
every subject as described in [16]. By consensus, co-authors selected a priori regions for testing within each of four groups CBS/PSP [17], bvFTD, svPPA and naPPA [18-24]. Details of the regions and rationale for their selection are available in the Supplementary Information. See Figure for an overview of processing, the CSRS method and a visualization of the regions (1.C).
142
3

---

(A)
0.5 mm
DBPN
(B)
Sample
Upscale
Map
Unity
1 mm
3D
(C)
bvFTD
naPPA
SVPPA
CBS/PSP
Figure (A) Image processing begins with raw MRI, extracts the brain, labels cortical regions, labels medial temporal lobe regions and labels deep brain regions. (B) The CSRS method is used, here, to upsample data by factor isotropically; the sketch the algorithm provides an example of how two nearby regions would flow through the method and be stitched back together at high resolution. (C) The impact SR on quantifying neurodegeneration assessed on a priori regions that are specific to each clinical diagnostic group; all regions bilateral except for svPPA which uses only left hemisphere cortical and medial temporal labels.
144
2.1
Concurrent super resolution and segmentation methods
145
CSRS uses, as sub-algorithm, three dimeessimani and multi-output version of the neural network architecture defined by the 2D deep back projection network (DBPN) [5]. The DBPN is uniquely relevant to medical imaging in that is perhaps the first published SR method that integrates the
147
148
caansampling-upsmmiiin error (i.e. residual layers) as feature map. This novel architecture may prevent feature hallucination and constrain the high-resolution image to maintain features that are consistent with the low-resolution input. We extend the 2D DBPN to 3D MRI data by, first, translating 2D convolutions, padding, striding and other relevant parameters to 3D. To generalize the architecture further, we allow options for not only convolutional upsampling (transposed convolution) but also nearest neighbor (or linear) interpolation layers at the user's ' choice. Lastly, we implement flexible choices of input channels, the number of residual layers (backprojection) layers and the number of outputs. This 3D DBPN network implementation is available within R and python. All parameters were the same the published work [25] (though in translation to 3D) with the exception of the number of back projection layers which has large impact on the number of parameters. We reduced number of backprojection layers caused by working with large 3D images and limited GPU resources (all GPU computations in this work were implemented with Nvidia V100s locally).
149
150
152
153
154
155
156
158
(16,264.322 parameters) due to the memory limitations
160
181
Efficient computational strategy is essential for CSRS to be applied to large 3D images (a brain image may contain 10 million voxels) when CPUs and RAM are limited. As such, local patch-work strategy is necessary. Sampling, upsampling, mapping and unification (SUMU) are the common steps needed for not only training but also inference. "Sampling" decides the form of the input data: full images (not used here), image patches (used here in training) or anatomical image regions (used here low-resolution data to higher-resolution output. Mapping compensates for shape or intensity distortion. Finally, "unification" an ensembling or merging step that brings together several sub-estimates of an SR image into single joined (final/full) SR image. We detail each of the 4 components below.
162
163
164
185
186
inference). Upscaling determines the core approach to transferring
188

---

170
Sampling: We choose patch- based model for training as these can easily be applied to input data with different resolutions and fields of view. second reason for patch-based modeling is that a candidate network does not need to learn the full scope of image variation. This results in shallower and faster to train networks that fit more easily onto readily available GPUs. The choices made during sampling step define the feature basis set. Because prior super-resolution competitions suggest larger patches lead to better performance, we choose the largest patches that would permit efficient batch sizes of 4 (64x64x64). An additional ad hoc support for this choice is that cortical features are relatively well-resolved in sub- Im training images when voxel cubes of this sized are used. However, there is no direct evidence that this size of patch domain is optimal for this problem.
171
172
173
174
175
177
178
179
Upscaling: is done with the CSRS's DBPN architecture using nearest neighbor interpolation for the upsampling layers. The software interface to CSRS also allows the user to optionally employ standard linear (tri-linear) interpolation. We use the linear option as a reference evaluation studies below.
180
181
182
183
Mapping: may be used to compensate for distortions in the image shape or intensity space. Because each patch is scaled independently on training data (to have an intensity range -12.5 to 127.5), the output of the PSR upsampled image intensities must be mapped back to the original quantitative space. This is performed by directly comparing the output of the PSR upsampled patch/region to the original data upsampled by nearest neighbor or linear interpolation. As such, we can accurately retain quantitative intensity data at the original scale/units with minimal distortion and/or stitching artifacts.
184
185
186
187
188
189
Unification: this is general term that, here, refers the algorithm that is used to derive a single CSRS image and multi-label segmentation from multiple CSRS sub-images (not necessarily isotropic patches in training). In 2D, multiple input images are typically generated from single input by "augmentation" e.g. random flipping, translation, etc thus allowing a practitioner to gain multiple "votes" about how the SR image should appear at any given voxel. Such a step is used in most PSR competitions to reduce aliasing or artifacts and may involve averaging, sharpening or more complex modeling such as joint intensity fusion, multi-channel deep learning or other ensemble methods.
190
192
193
194
195
196
Due to the high memory and computation cost of running CSRS on 3D images, we instead apply CSRS either sub-regions of interest or, when a full Tlw brain image is desired, each hemisphere.
197
198
The unification step then maps each local patch intensity range back to the original MRI range and then joins the sub-regions back together to complete the SR reconstruction. Augmentation can be employed beyond this but at substantial increase in computation time (e.g. 10x to see meaningful gains due to augmentation).
199
200
201
202
Loss functions for CSRS: We employ a loss function that seeks to balance reconstruction error (intensity difference, abbreviated here as R), edge preserving denoising (total variation, abbreviated as TV), perceptual quality (based on VGG or ResNet) and segmentation overlap (Dice, abbreviated as D). Each of these terms can be up or down weighted to control the network's 's performance where mean squared error (L2 intensity error) leads to smoother results, LI (or total variation) provides denoising and the perceptual loss yields more natural appearing output textures and shapes. The
203
204
205
206
207
208
Dice loss term seeks to minimize distortions in the shape of segmentation objects on the output of CSRS. The Dice loss is only applied to the second output channel of the network which uses a sigmoid activation function appropriate for probabilistic/bnnrry data. We refer to CSRS trained with
209
210
specific combinations of these losses by concatenation of the abbreviations above. For example, 211
CSRS.R. TVD.D. Resó refers to network trained with reconstruction loss, TV regularization, Dice loss and the 6th layer of the TesQRResNet for perceptual loss.
213
214
Recent research demonstrates that deep learning models trained on large-scale object detection reference datasets (e.g. imagenet) encode a feature space that may mimic human perception [26]. Such perceptual spaces typically arise from the activations that occur within the layers of convolutional networks trained on massive classification datasets. Here, however, we compare a (mapped to 3D as described before) to those defined by the TewQRResNNe. From Ti TIWQRResNer we choose two different deep layers that have similar numbers of parameters to the 3D version of the VGG19 block2_ conv2 network: res conv_ conv_bloce block_6 (the 2nd convolutional block) and res_ conv_ conv_block. ___1 (the 7th convolutional block). This allows us to compare perceptual metrics based on either pseudo-3D VGG19 or our intrinsically 3D cass__oovbblll
215
216
217
218
standard VGG based perceptual space (block2_con/))
219
220
221
223
choices.
224
Quantiiicatinn of medical images requires a high degree of faithfulness to the input data. "Halluci- nated" features are undesirable. As such, our baseline loss function focuses on reconstruction error
225
5

---

A
(B)
Cortex (volume, thickness)
Deep brain mid-brain (volume)
Medial temporal lobe (MTL) (volume)
Best model CSRS applied to three categories of anatomy where row (A) is the original resolution (OR) and segmentation and row (B) the output of CVR..R.TV.D.Re66.
Figure 2:
226 and TV for both intensity and segmentation images. We then add perceptual and Dice losses for further comparison. If we denote as the estimated super-resolution, I, as the estimated segmentation from the sigmoid output channel, the real high resolution image, J, as the real high resolution segmentation, then the final loss function that we optimize is:
227
228
||/-]|| wt +|1,, J. w +TV (), wV J.) W -I
J.) wa
230
where the term || indicates the euclidean norm, (V.,)) indicates the total variation norm (which provides denoising), Wr.t.fdd (superscripts for intensity segmentation) indicates term-specific scalar weight and (.)() indicates perceptual feature map. The weight terms can tuned for performance and application area given objective and quantitative evaluation metric. We initially manually tuned the training of a DBPN model with only the reconstruction metrics ( J]l w
232
234
11, 122 we with W 5e 4 and w 1) using adam optimizer and learning rate 5e-5. We then set weights relative to the value the reconstruction error after convergence such that: the TV loss is roughly 2/3 the reconstruction term (R); the perceptual loss is roughly 3x R; the Dice loss is
236
roughly equivalent to the perceptual loss. This strategy, based on our task-specific goals, enables us compare models consistently and add/subtract terms without extensive weight optimization.
238
240
Computation and inference: All models were implemented with tensorflow. The computation to modern computational platform)
double magnification for single Tlw takes (generally between 10 and 40 minutes. Results are computed region-wise over the set segmentation labels where CSRS is run on each cropped label and its associated intensity. When multiple regions used (as is done here), then results are stitched back together while using a linear mapping back to the original intensity space and a arg_mas- max operation to define the hard segmentation labels at every voxel the stitched, joint intensity/probabiiity double magnification space. See Figure 2 for example result of CSRS applied to the variety of brain regions in this study. Figure shows zoomed visual comparison of the impact on intensity and the lack of stitching artifacts.
242
244
246
248
2.2
Quantification of CSRS impact segmentation and intensity in ground truth data
250
Evaluation of PSR results on simulated paansssmleedupssapppee data does not constitute real world conditions. However, for reference, we include evaluation results based on an independent set of
251
252
labeled brain images [27]. For these images, we downsample with nearest neighbor interpolation and upsample with linear interpolation (for the intensity) and a "generic label" interpolation that designed for multi-label images [28] thereby allowing us report standard metrics of Dice overlap,
253
255
PSNR and SSIM to complement our study of brain atrophy detection. Figure 4 demonstrates example results illustrating this component of our evaluation.

---

(A) Original
D
(5) R RB
(B) Lineer
(D) R VG
(F) R RV1 D
Figure 3: Comparison of CSRS with different loss functions original resolution and linear upsam- pling. The bold (panel E) is the best performing model according to quantitative criteria. However, visual differences between the perceptual models (D,E,F) are not easy discern.
CSRS.R.TVD Res2 Dice minus linear Dice linear Dice
02
(A) Ground truth
(8) Linear
(C) R.R.S.R.TV....O..
(D)
Figure 4: Panel (A) shows the original mmm resolution ground truth image and its segmentation.
Panel (B) shows the impact of linear/generic label upsampling of ground truth data artifically downsampled to 2mm3 Panel (C) shows a CSRS result where other models are visually similar to this. Panel (D) demonstrates that all regions improve with CSRS (all differences >> 0) and that regions with lower Dice overlap under the linear/generic label model improve more when upsampled with CSRS,
257
2.3
Quantification of effeet sizes in frontotemporal disorder atrophy
258
The frontotemporal disorders produce profound and debilitating effect on patients with concomitant, sympiom-related atrophy. Measuring this atrophy is critical to detecting the effects, for instance, of disease modifying therapies that may slow atrophy. Such measurements are challenged by low resolution and this challenge is compounded by the degeneration process itself.
259
260
We statistical modeling to determine if CSRS can mitigate the known limitations of resolution atrophy measurement. We adopt an interpretable mixed effects modeling approach (1mer)[29) to estimate effect sizes per brain region, per diagnostic category and per resolution/CSRS model. The baseline performance determined by the effect sizes estimated on the original resolution (OR) data. We estimate effect sizes following [30, 31]. Better methods, under this design, should more reliably identify disease-related atrophy which will be reflected in increased effect sizes for given set of priori
263
265
257
288
regions. The model for the region of interest (ROL) is:
ROI se Ages + Ser + BV, + DX + AT + DX + +1IID),
289
with (1|ID) representing subject-specifie random effect, Ages is the subject's 's age at the first visit, BV, is the first visit brain volume, DX is the diagnosis for the subject, AT is the change in time since baseline and the AT* DX represents an interaction between time and diagnosis. The ROI; represents the volume for all regions. However, for cortical regions, we also use the region's thickness measurement second outcome (as this is a standard measurement in morphometry of the human cortex). We estimate effect sizes for cross-sectional effects via the model's 's parameter fit for the diagnosis (DX) term; we estimate longitudinal effect sizes via the parameter on the interaction term.
270
272
274
7

---

Table 1: Summary of results where the comparison of the model impact on effect size is computed by bootstrapped (n=1000) paired t.tstt. The number of pairs 274 (see Table for further break- down by category). CSRS losses are abbreviated as R-reconstruction TV=total variation, D=dice, VGG=VGG19 pseudo 3D features, Resó is from the 6th layer of Tl WORResNet and Res21 the 1 layer of TI TeeRReeeee.. srmeanES indicates the mean effect size for the model averaged over all a priori regions; boot.95ci the 95 percent confidence interval for the improvement effect size due to the model. t represents the i-statistic and boot.p represents the bootstrapped p-value for the significance of the improvement effect size. Columns psnr and ssim show the standard PSNR and SSIM values for image for which we have ground truth high-resolution intensity and segmentation. The dice columns show the mean and standard deviation of the Dice overlap between ground truth and the upsampled simulated data with each model, estimated over all regions. Best = bold.
Model
srmeanFS
bont95ci
boot.p
psnr
ssim
dice.mean
dice.sd
OR
0.359
070
NA
NA
NA
NA
NA
NA
Linear
0,468
40. N.10067 -0.08163
18.61
0
40.6
0.996
0.769
0.047
CV
0.574
0.01124/0.00 70.01854
7.96
0
42.0
0.997
0.884
0.031
CSRS.R.VI
0.582
0.01868 / 0.0273
10.38
0
41.8
0.997
0.883
0.031
CSRS.R I VVVGG
0.581
0.01775 0.02559
10.76
0
42.0
0.997
0,885
0.031
CSRS.R. TV.D VGG
0.577
0.01403 / 0.02209
8.86
41.9
0.997
0.884
0.031
ASRS.R.TVRR..
0.572
0.009741 0,01665
7.49
42.4
0.997
0.886
0.031
CSRS.R.VV. TVD...es
0,588
K.02497700330
14.02
42.4
0.997
0.885
0.032
CSRS.R. T. TVRRs22
0,377
0.01462 /0.02143
10.40
42.3
0.997
0.884
0.031
CSRS.R.IV...Res/
0,581
0.0181470.02627
10.59
42.3
0.997
0.887
0.03
bwFTD
croco sectional
longitudinal
avPPA
Contical thickress
Cortical volume
Cortical thicknoss
Cortical
Medial tamporal
9R 11/0 OR offect 5400
ebos
5:05
effect size
offect
CBS/PSP
Cortical thicknsess
Cortical volume
Cortical thicknesss
Cortical volume
Deep brain
BR minu OR- 220
3.3
offect
Figure 5: Bland-Altman plots for model CSRS. D.V.S...V...RRe.... demonstrate variability in the per- formance by type of anatomy and by diagnostic grouping with some individual points generating substantially greater % improvement than suggested by the overall trend. Similarly, a few points show decreased performance relative to OR.
3 Results
276
277
Table summarizes overall results where we show original resolution and results from linear upsampling, as baseline, and compare to eight variants of CSRS. Two of these do not use perceptual metrics. The remaining six add or subtract Dice loss and each of our candidate perceptual losses. Table 1 shows both the aggregate impact of model effect size estimates in the neurodegeneration data as well as intensity similarity (reconstruction) and Dice overlap in the ground truth data. Dice overlap (a measure that varies between zero and one) improves by margin of 0.11 I to 0.3 (95% CI bootstrapped percentile confidence interval, p < le 16. See Figure 4.
279
281
283
284
Table 2 focuses on the two perceptual models with the greatest improvement from original resolution as assessed by pairwise (-test. It breaks down the effect size results in relation to which type of effect size is being analyzed (cross-sectional or longitudinal) and by brain region diagnostic grouping. Relatedly, Figure shows Bland-Altman style plot that demonstrates, for the CSRS.R.TV.D.R.
285
287
model, the range of effect size changes due to CSRS across all 274 measurement points.
8

---

Table 2: Summary of results for the two best perceptual models broken down by anatomical class, type of predictor (longitudinal or sectional) and diagnostic groups. The column indicates the number of samples used in the statistical testing. The codes in the AnatClass column are: CtxV cortical volume; CixT cortical thickness; MB deep brain (for CBS/PSP); MTL- - medial temporal lobe (for svPPA). The columns that have non-NA DX2 means that both DX and DX2 groups were aggregated the computation of the bootstrapped paired I-test for the given group of anatomy.
Model
AnaiClass
LsLong
DX
DX2
srmeanES
boot.95c
boot.p
ISRS.R.VVVGG
All
Both
NA
NA
274
0.581
0.01775 / 0000259
10.763
0.0000
CSRS.R.VVV
CIVV
Cross
byFTD
naPPA
28
0.826
0.0071537 0.01626
4,936
0.0000
CSRS.R.IV.VGE
CRT
Cross
bvFTD
naPPA
28
1.016
-0.009547 / 0.011006
0.033
0,9769
VVVGG
MB
Cross
CRSIPSP
NA
8
0.600
0.036837 1208
3.458
0.0246
IV.VG
MTL
Cross
SvPPA
NA
7
1.003
0.02801 / 70006981
4.190
0.0112
CSRS.R.IV.VG.
CIXV
Long
bvFTD
naPPA
28
0.645
0.01848/ / 00003313
6.719
0.0000
IVVG
CITT
Long
bvFTD
naPPA
28
0.527
0.03521 70.05297
9.445
0.0000
CSRS.R. VV.GG
MB
Long
CBSIPS
NA
8
0.201
0.01409/ 0.00344
3,924
0.0110
CVVS
MTL
Long
svPFA
NA
7
0.701
-0.01997 0.002113
-1.309
0,2652
CSR.R..IV.D.Res
AlI
Both
All
NA
274
0.588
0.02497 / 70.0331
14.021
0.0000
IV...R.IVD..Res
CRVV
bwFTD
naPPA
28
0.831
0:01999 0002282
6.573
0.0000
CSRSRR I Reso
CIXT
Cross
byFTD
paPPA
28
1.033
40,002966 / 0.01833
1.394
0.1604
R.DD
MB
Cross
CBSPS
NA
8
0.589
0,02166 / 71.1131
27718
0.0454
CSR.R.IVDDDRess
MTL
Cross
svPPA
NA
1.044
0.02782/ / 70.07533
4.021
0.0102
CSRS.R. R.D.D.
CtXV
Long
bvFTD
naPPA
28
0.658
0.032037 7 70,04756
9.751
0,0000
CSRS.R. Res6
CtXT
Long
bvFTD
naPPA
28
0.545
0.05421 / 0000699
15.151
0.0000
CSRS.R. TV.D.Rest
MB
Long
CRS/PSP
NA
8
0.198
0.006017 0.04464
2.233
0.0166
R.RS.R.TV.D...
MTL
Long
svPPA
NA
7
0.704
/0.0177/0.01214
-0.369
0.7511
289
4
Discussion
The PSNR and SSIM improve similarly across all CSRS models and do not substantively differentiate performance. Dice overlap is consistently superior than linear upsampling across all models but shows little difference between models with perhaps small advantage for the ResNet features. Greater stratification may be seen when looking at results that relate to quantifying the phenotypic hetero- geneity of brain atrophy in frontotemporal spectrum diagnostic groups. Model S.RS....V.DDRR.. stands out under this criteria with Table suggesting that the majority of the improvement arises for cortical measurements, particularly longitudinally. Performance improvements are not, however, perfectly consistent. Figure 5 shows that CSRS augments effect size in the large majority of regions (some greatly so) but few regions are subtly better at OR. Additional discussion of performance implications with respect to individual regions and diagnoses is in supplementary information.
291
293
295
297
299
The extension of PSR to 3D raises opportunities as well as challenges. Parameter exploration is fundamentally limited because training a model on our patch dataset for epoch takes over 12 hours (we trained each model for 2 epochs until convergence). Other architectures than DBPN may perform better with CSRS such as ESRGAN [32] or, potentially, methods with stronger modality specific priors on the convolutional kernels [33]. Specifically, fast-training, fewer parameter models may ease some of the computational burden and facilitate more parameter exploration.
301
302
303
305
306
CSRS performance is fundamentally limited by the quality of its segmentation inputs. It may be more beneficial to develop new methods that operate at high resolution (HR)- adding substantial computational cost if the goal is to take advantage of HR features or that take advantage of intrinsically HR ground truth data. The primary barrier to such an effort is the current lack of HR ground truth labels for neuroimaging and in particular for neurodegenerative disease. Moreover, most methods embed resolution assumptions in their own processing choices and optimize for these choices. As such, CSRS bridges performance gap with practical solution readily available today.
307
309
310
311
312
313
Retooling existing methods and segmentation labels for HR (e.g. 7T MRI) is costly both computation- ally and in terms of the effort of human experts due to the already high volume of 3D neuroimaging.
314
315
We demonstrated that CSRS, in most of its variants, leads to significant performance improvements
over our reference of original resolution ( (Imm²) image processing and ground truth labels. Because
316
317
CSRS operates on existing images and labels, new HR method and segmentation development is
not required. Thus, CSRS may be used to improve existing ground truth datasets and existing
319
processed data, today. However, comparison to other and/or larger real world datasets is needed to
help determine the extent to which our results may be deployed to new data without concern.
9

---

References
321
322
Mulder MJ, Keuken MC, Bazin PL., Alkemade A, Forstmann BU. Size and shape matter: The impact of voxel geometry the identification of small nuclei. PLoS ONE. 2019. attpps//doo org/10. 1 pone. 0215382
324
325
2. Veitch DP, Weiner MW, Aisen PS, Beckett LA, Cairns NJ, Green RC, et al. Understanding disease progression and improving Alzheimer's disease clinical trials: Recent highlights from the Alzheimer's disease neuroimaging initiative. Alzheimer's ' & Dementia: the Journal of the Alzheimer's Association.
326
327
328
2019:15:106-5..
329
3. Boxer AL,, Gold M, Feldman H, Boeve BF, Dickinson SL-J, Fillit H, et al. New directions in clinical trials for frontotemporal lobar degeneration: Methods and outcome measures. Alzheimer's & dementia: the journal of the Alzheimer's Association. 10:0016::31-10
330
4. Blau Y, Michaeli T. The prrception-distortion tradeoff. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 6228-6237, 2018. 2017. https://doi. //tg:////oooo//00/0/
333
11 1109/CVPR. 2018 00652
335
Haris M, Shakhnarovich G, Ukita N. Deep back-projectio networks for single image super- resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020; 43(12):4323- 4337.
337
338
6. Agustsson E, Timofte R. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops 2017. pp. 126-135.
339
340
7. Gu S, Danelljan M, Timofte R. Haris M, Akita K, Shakhnarovic G, et al. AIM 2019 challenge image extreme super-resolution: Methods and results. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Scoul, Korea (South): IEEE, pp. 3556-64 64.
342
344
Blau Y, Mechrez R, Timofte R, Michaeli T, Zelnik-Manor L. The 2018 PIRM challenge on perceptual image super-resolution. In: 2018 Proceedings of the European Conference on Computer Vision (RCCV) Workshops. pp. 334-55.
346
347
9. Tian Q Bilgic B, Fan Q, Ngamsombat C, Zaretskaya N, Fultz NE, et al. Improving in vivo human cerebral cortical surface reconstruction using data-driven super-resolution. Cerebral cortex (New York, NY : 1991). 2021;31:463-82.
348
349
10. Glasser MF, Smith SM, Marcus DS, Andersson JLR, Auerbach EJ, Behrens TEJ, al. The human connectome project's ' neuroimaging approach. Nature Neuroscience. 2016; 19(9): pp. 1175-1187.
351
11. Elam IS, Glasser MF, Harms MP, Sotiropoulos SN, Andersson JL. Burgess GC, et al. The human connectome project: A retrospective. Neurolmage. 2021:244:118543
353
354
12 Zhang K. Hu H, Philbrick K. Conte GM, Sobek JD. Rouzrokh P, et al. SOUP-gan gan: Super- resolution mri using generative adversarial networks. Tomography (Ann Arbor, Mich). 2022;8:905-
355
356
19.
13. Shan H, Zhang Y. Yang Q, Kruger U, Kalra MK, Sun L, al. 3-d convolutional encoder network for low-dose ct via transfer learning from a 2-d trained network. IEEE Transactions on Medical Imaging. 2018. https://doi. org/ 10. 1199/MMI, T01.2018........... 2832217
358
360
14. Avants B, Greenblatt E. Hesterman J, Tustison N. Deep volumetric feature encoding for biomedical images. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2020.
362
383
15. Zhang R, Isola P. Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as perceptual metric. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2018.
364
365
16. Tustison NJ, Cook PA, Holbrook AJ, Johnson HJ, Muschelli Devenyi GA, et al. The antsx ecosystem for quantitative biological and medical imaging. Scientific reports. 10:0;81
367
17. Illán-Gala I, Nigro S, Vande Vrede L., Falgås N, Heuer HW, Painous C, et al. Diagnostic accuracy of magnetic resonance imaging measures of brain atrophy across the spectrum of progressive supranuclear palsy and corticobasal degeneration. JAMA network open. 2022:5:
369
10

---

18. Whitwell JL. FTD spectrum: Neuroimaging across the ftd spectrum. 2019:187-223,
19. Whitwell JI. Josephs KA. Neuroimaging in frontotemporal lobar Degenerationn-prddicinnn molecular pathology. 1012:8:131-42
373
20, Binney RJ, Pankov A, Marx G, He X. McKenna F, Staffaroni AM, et al. Data-driven regions of interest for longitudinal change in three variants of frontotemporal lobar degeneration. Brain and behavior. 2017:7:c00675.
375
377
21. McCarthy J, Collins DL, Ducharme S. Morphometric mri as a diagnostic biomarker of fron- totemporal dementia: A systematic review to determine clinical applicability, Neurolmage Clinical.
378
379
201 8:2::685-99
380
22. Gunawardena D, Ash S, McMillan C, Avants B, Gee J, Grossman M. Why are patients with progressive nonfluent aphasia nonfluent? Neurology. 2010:75.
382
23. C.T. M, T, B.B. A, P.A. C, E.M. W, E. S, et al. Genetic and neuroanatomic associations in sporadic frontotemporal lobar degeneration. Neurobiology of Aging. 2014.
24, Massimo Powers C, Moore Vesely L, Avants Gee al. Neuroanatomy of apathy and disinhibition in frontotemporal lobar degeneration. Dementia and Geriatric Cognitive Disorders.
385
386
2009. https://doi, org/ 10. 1159/000194658
387
25. Haris M, Shakhnarovich G, Ukita N. Deep back-projection networks for super-resolution, In: Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition. 2018.
389
26. Zhang R. Isola P. Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as perceptual metric. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2018.
391
393
27. Klein A, Tourville 101 labeled brain images and consistent human cortical labeling protocol. Frontiers in neuroscience. 2012;6:171.
28. Schaerer Roche F, Belaroussi B. A generic interpolator for multi-label images. 2014. https /doi org/ 1...54294//r1111.
396
397
29. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using Ime4 bates
396
journal of statistical software. Journal of Statistical Software. 2015;67.
389
30. Brysbaert M, Stevens M. Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition. 2018;1.
400
401
31. Ben-Shachar M, Lüdecke D, Makowski D. Effectsize: Estimation of effect size indices and standardized parameters. Journal of Open Source Software. 2020:5.
405
Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, et ESRGAN: Enhanced super-resolution generative adversarial networks. arXiv e-prints. 2018;arXiv: 1ei.Xxv:1899002199
405
33. Bell-Kligler Shocher A, Irani M. Blind super-resolution kernel estimation using an internal-gan.
406
In: Advances in Neural Information Processing Systems. 2019.

---

Checklist
407
408
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
410
contributions and scope? [Yes]
411
(b) Did you descrihe the limitations of your work? [Yes]
412
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
414
415
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
417
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We provide the data, in supplementary material, that is needed to generate the tables shown in the paper. The raw data is publicly available but we do not have permission to redistribute these data. However, do provide the ability to reproduce key results in the Tables of the main manuscript via supplementary information.
420
422
424
425
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
428
427
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See beginning of Methods section and CSRS methods section.
430
431
4. If you are using existing assets code, data, models) or curating/releasing new assets...
433
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] in supplemental information.
(c) Did you include any assets either in the supplemental material or as a URL? [Yes]
436
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] in supplemental information.
437
(e) Did you discuss whether the data you are using/curating contains personally identifiable
439
information or offensive content? [Yes] in supplemental information.
440
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
442
443
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
444
445
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "s1s4idxrD-", "reviewer": "Reviewer_hYgi", "summary": "This work developed a super resolution method for 3D neuroimaging and evaluate its performance in detecting brain changes due to neurodegenerative disease. It trained on 3D brain data to upsample both the raw intensity image and associated segmentation labels. The method was mainly based on the 2D deep back projection network (DBPN) [5], which was extended to three dimension and multiple output. The method was evaluated in downstream clinically relevant signal detection problem: quantifying cross-sectional and longitudinal\nchange across a set of phenotypically heterogeneous but related disorders that exhibit known and differentiable patterns of brain atrophy. ", "questions": "See the detailed comments above.", "limitations": "No.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n-- This work conducted the evaluation of super resolution in the downstream clinically relevant tasks, e.g., quantifying cross-sectional and longitudinal change across a set of phenotypically heterogeneous but related disorders.\n\nWeakness:\n-- This work overall has limited methodology contribution. \n\n-- The organization of this work is not clear and hard to follow for some sections.\n\n-- This work fits more to a dedicated medical imaging conference or journal.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "mtNr7thIqrf", "reviewer": "Reviewer_aKcw", "summary": "This paper outlines a new method for enhancing 3D neuroimaging resolution and improving image segmentation using \"perception super resolution\" in a technique labelled \"concurrent super resolution and segmentation\" (CSRS). The authors evaluate the effectiveness of the technique on publicly available clinical datasets from the Human Connectome Project, the Parkinson's Progression Markers Initiative, and the Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & 4-Repeat Tauopathy Neuroimaging Initiative (4RTNI). They test this method in quantifying changes across cross-sectional and longitudinal MRI datasets for different neurodegenerative diseases. The results, however, only report comparisons of the new technique to the original-resolution model using bootstrapped t-tests, which are not corrected for multiple testing. The implications are very briefly discussed. ", "questions": "I do not have any questions", "limitations": "The authors do not address the limitations or potential negative social impact of their work. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I do not comment on paper originality as it is a flawed measure of paper quality. \nThe overall quality of the paper is moderate; while the technique is interesting, there is insufficient detail about preprocessing of the MRI data to replicate the results, and the methods are not communicated very clearly. In addition, more of a clinical and/or neuroscience perspective in the experimental design would have been useful. The Discussion is overly brief and lacking in depth or detail. The Supplementary Materials are quite poorly organized and do not meet NeurIPS communication standards. However, the technique itself does seem very promising, and with clearer communication, this could be an interesting and important development. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fuwrgKvtUu", "reviewer": "Reviewer_CkcC", "summary": "This work introduces a new perceptual super resolution method for 3D brain image segmentation. The method uses a carefully designed up-sampling and a novel loss function to obtain the desired performance, and this method is evaluated based on a clinical relevant metric due to the lack of high resolution ground truth data. The authors find that the proposed method consistently improves the ability to detect regional atrophy both longitudinally and cross-sectionally in five relevant diseases.", "questions": "Please refer to the \"weakness\" part.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- This work addresses an important problem in the field of medical imaging, and the authors have demonstrated deep understanding and expertise in this field.\n- The method is discussed in a thorough manner, and important decisions are well justified. Truly quality work!\n- Rigorous experiments to evaluate the proposed method's performance (it's great to see CI evaluations)\n\nWeakness:\n- This work would be of a stronger stance if its originality is more well-justified: 1) the paper writes that \"...three perceptual loss functions for PSR, two of which are new\", while the neural network and the loss functions used (reconstruction error, TV, perceptual loss, dice) are not unheard in the field of medical imaging. The authors might elaborate which parts of the loss functions are new. 2) While the proposed evaluation paradigm is, in a sense, new, I would like to see a deeper discussion of why this is new (e.g., is this a novel way of seeing the problem of lacking SR ground truth label?)\n- The contribution bullet states that \"demonstration of the impact of loss choice on performance differences in improving detection power in [population studies] of neurodegenerative disease\". The presented results haven't directly shown the power of the proposed method on a population-scale. The authors might elaborate more about the stated population-level effect of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d0QHJnW5xx4", "reviewer": "Reviewer_Uzm9", "summary": "This paper proposed a 2x (from 2mm to 1mm) super-resolution method for 3D medical image volumes and the corresponding segmentation mask for better brain diagnosis. The proposed method is a natural extension of the 2D super-resolution model called deep back-projection network (DBPN). Compared with linear upsampling and nearest-neighbor upsampling, the proposed method achieved a better dice score as well as a better visual quality. ", "questions": "Is this the first super-resolution 3D model in this area? If not, can you make a comparison between the proposed method and the existing methods?", "limitations": "As stated in the weakness, the method is not novel to me and the experimental comparison is not satisfactory for a NeuroIPS paper. The paper lacks a thorough description of related work and a better comparison with other existing methods other than linear/nearest-neighbor upsampling.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThis paper provides good implementation/technical details to illustrate the problem, the method, and the experiments. It also provides a good visual-based comparison to interpret the method's performance. It is interesting to consider upsampling both the image along with the segmentation mask.\n\n\nWeakness: \nThe method proposed in the paper doesn't seem to be novel given the fact that it is a natural extension from 2D to 3D based on the deep back-projection network (DBPN). Though the paper proposed a few loss functions to adjust the model's objective, the performance does not show a significant difference between different combinations of loss functions. \n\nMoreover, the paper doesn't provide a good description of the related work and the main comparison is done between the variants of the proposed method and the trivial methods (linear/nearest-neighbor upsampling).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VlNE9fo5iz4", "reviewer": "Reviewer_kpkJ", "summary": "The authors present an extension of a 2D perceptual super-resolution architecture extended to 3D within neuroimaging. The authors additionally extend the network to simultaneously produce up-sampled segmentation maps. The upsampled segmentation maps are evaluated against the ground truth, as well as against detection power across a variety of frontotemporal disorders. The effect of different loss functions used in training are also evaluated against prediction power. Whilst the super-resolution techniques improved standard evaluation metrics (PSNR & SSIM) when compared against the low resolution data, these metrics did not significantly improve when compared against models trained with the different loss functions used. Prediction power for frontotemporal disorders was improved in many regions when using the super-resolution models, however some regions suffered in prediction power when super-resolved.\n\n", "questions": "I have no questions at this point.", "limitations": "The authors have addressed the limitations of their work in the conclusions section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this paper is clear and concise, and forms a well constructed manuscript. The background information and methods are extensive, and are presented in logical, clearly defined, sections. The overall study is fairly in depth for the given research question, and presents a nice analysis of the downstream effect of segmentation super-resolution within a disease model. The authors include the relevant previous work done in this area, and appropriately outline their contribution to this field.\n\nThe biggest limitation of this work I feel is the limitation of impact in some of novel contributions outlined. Specifically, extending a model from 2D to 3D, whilst relevant within neuroimaging data, is not particularly impactful. Additionally, adding segmentation maps as an additional output channel does improve user convenience, and likely a computational efficiency gain, but is ultimately not overly impactful as the segmentation maps could be re-calculated with the inferred high-resolution dataset. Overall however, it is my opinion that the strengths do clearly outweigh the weaknesses in this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UP-MV6Mth_", "reviewer": "Reviewer_Ho1b", "summary": "The paper presents a new methodology for concurrent super resolution and segmentation (CSRS) on 3-D volumetric MRI data to consistently upsample both an image intensity channel and associated segmentation labels. To this end, a primary contribution is in adapting perceptual super resolution frameworks designed for 2-D data to 3-D data within the 2D deep back projection (DBPN) network.\n\nSince the ground truth high resolution images may not be present for a given dataset, the authors propose an indirect evaluation via the quantification of cross-sectional and longitudinal change in diseased cohorts. Specifically, they choose a set of phenotypically heterogeneous but related disorders that are associated with known patterns of brain atrophy. Their experiments examine the effect of various choices in loss function in terms of identification of neurodegenerative diseases within the Human Connectome Project (HCP), Data: Parkinson’s Progression Markers Initiative (PPMI) , Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & and-Repeat\n123 Tauopathy Neuroimaging Initiative (4RTNI) datasets.", "questions": "1. Section 2.1 : \"We then set weights relative to the value of the reconstruction error after convergence such that: the TV loss is roughly 2/3 the reconstruction term (R); the perceptual loss is roughly 3x R; the Dice loss is roughly equivalent to the perceptual loss. This strategy, based on our task-specific goals, enables us  to compare models consistently and add/subtract terms without extensive weight optimization.\"\n\nHow did the authors arrive at this heuristic?\n\n2. Formatting issues\n\n(a) Oddly large spacing after section 2 heading \n\n(b) The plots in the paper are a bit blurry and tables are included within figures. A suggestion would be to replace these with better resolution images and use the default latex package for displaying the tabular data.\n\n3. Discussion \"we trained each model for 2 epochs or until convergence\", does this imply that the maximum number of epochs used is 2?", "limitations": "The main limitations discussed in section 6 including computation and quality of annotations. Given that the framework is so computationally expensive that 1 epoch takes 12 hours seems to greatly limit the practical utility. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS: The clinical problem that the paper examines, concurrent super resolution and segmentation is interesting, as is the use-case  of tracking brain atrophy within neuro-degenetative diseases. The proposed approach to extending existing Perceptual Super Resolution techniques is interesting, though a bit incremental on the technical front.\n\nWEAKNESSES:\n\n1. The evaluation section is rather poorly explained and hard to follow. Several points are unclear and the main arguments are not very convincing in light of the quantitative results\n(a) Some key details such as dataset splits for training, testing, validation are not clearly mentioned.\n(b) It is unclear which dataset/disorder (all datasets?) the results in Table 1 and 2 and figures correspond to.\n(c) The differences across different loss functions and configurations in Table 1 are rather minor (third decimal place in terms of effect sizes, dice, psnr). It is unclear whether these improvements are consistent across dataset splits.\n(d)  Table 2 is really hard to parse, lacking in description and in general confusing. As per my understanding, CSRS.R.TV.VGG vs the proposed framework should be compared across the same anatomical class. If so, the differences in effect sizes appear very minor (third decimal place) for several comparisons.\n\n2. If my understanding is correct, the baseline comparisons in the paper correspond to evaluation against ablated versions of the framework (various loss functions), evaluation in the original resolution and linear interpolation. Given that the authors listed a couple of recent approaches proposed in literature in Section 1, it is not obvious why they did not include these as baseline comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject