Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces a new perceptual super resolution method for 3D brain image segmentation. The method uses a carefully designed up-sampling and a novel loss function to obtain the desired performance, and this method is evaluated based on a clinical relevant metric due to the lack of high resolution ground truth data. The authors find that the proposed method consistently improves the ability to detect regional atrophy both longitudinally and cross-sectionally in five relevant diseases.
strengths_and_weaknesses: Strength:
- This work addresses an important problem in the field of medical imaging, and the authors have demonstrated deep understanding and expertise in this field.
- The method is discussed in a thorough manner, and important decisions are well justified. Truly quality work!
- Rigorous experiments to evaluate the proposed method's performance (it's great to see CI evaluations)

Weakness:
- This work would be of a stronger stance if its originality is more well-justified: 1) the paper writes that "...three perceptual loss functions for PSR, two of which are new", while the neural network and the loss functions used (reconstruction error, TV, perceptual loss, dice) are not unheard in the field of medical imaging. The authors might elaborate which parts of the loss functions are new. 2) While the proposed evaluation paradigm is, in a sense, new, I would like to see a deeper discussion of why this is new (e.g., is this a novel way of seeing the problem of lacking SR ground truth label?)
- The contribution bullet states that "demonstration of the impact of loss choice on performance differences in improving detection power in [population studies] of neurodegenerative disease". The presented results haven't directly shown the power of the proposed method on a population-scale. The authors might elaborate more about the stated population-level effect of the proposed method.
questions: Please refer to the "weakness" part.
limitations: Yes.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper outlines a new method for enhancing 3D neuroimaging resolution and improving image segmentation using "perception super resolution" in a technique labelled "concurrent super resolution and segmentation" (CSRS). The authors evaluate the effectiveness of the technique on publicly available clinical datasets from the Human Connectome Project, the Parkinson's Progression Markers Initiative, and the Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & 4-Repeat Tauopathy Neuroimaging Initiative (4RTNI). They test this method in quantifying changes across cross-sectional and longitudinal MRI datasets for different neurodegenerative diseases. The results, however, only report comparisons of the new technique to the original-resolution model using bootstrapped t-tests, which are not corrected for multiple testing. The implications are very briefly discussed. 
strengths_and_weaknesses: I do not comment on paper originality as it is a flawed measure of paper quality. 
The overall quality of the paper is moderate; while the technique is interesting, there is insufficient detail about preprocessing of the MRI data to replicate the results, and the methods are not communicated very clearly. In addition, more of a clinical and/or neuroscience perspective in the experimental design would have been useful. The Discussion is overly brief and lacking in depth or detail. The Supplementary Materials are quite poorly organized and do not meet NeurIPS communication standards. However, the technique itself does seem very promising, and with clearer communication, this could be an interesting and important development. 
questions: I do not have any questions
limitations: The authors do not address the limitations or potential negative social impact of their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper presents a new methodology for concurrent super resolution and segmentation (CSRS) on 3-D volumetric MRI data to consistently upsample both an image intensity channel and associated segmentation labels. To this end, a primary contribution is in adapting perceptual super resolution frameworks designed for 2-D data to 3-D data within the 2D deep back projection (DBPN) network.

Since the ground truth high resolution images may not be present for a given dataset, the authors propose an indirect evaluation via the quantification of cross-sectional and longitudinal change in diseased cohorts. Specifically, they choose a set of phenotypically heterogeneous but related disorders that are associated with known patterns of brain atrophy. Their experiments examine the effect of various choices in loss function in terms of identification of neurodegenerative diseases within the Human Connectome Project (HCP), Data: Parkinson’s Progression Markers Initiative (PPMI) , Frontotemporal Lobar Degeneration Neuroimaging Initiative (NIFD) & and-Repeat
123 Tauopathy Neuroimaging Initiative (4RTNI) datasets.
strengths_and_weaknesses: STRENGTHS: The clinical problem that the paper examines, concurrent super resolution and segmentation is interesting, as is the use-case  of tracking brain atrophy within neuro-degenetative diseases. The proposed approach to extending existing Perceptual Super Resolution techniques is interesting, though a bit incremental on the technical front.

WEAKNESSES:

1. The evaluation section is rather poorly explained and hard to follow. Several points are unclear and the main arguments are not very convincing in light of the quantitative results
(a) Some key details such as dataset splits for training, testing, validation are not clearly mentioned.
(b) It is unclear which dataset/disorder (all datasets?) the results in Table 1 and 2 and figures correspond to.
(c) The differences across different loss functions and configurations in Table 1 are rather minor (third decimal place in terms of effect sizes, dice, psnr). It is unclear whether these improvements are consistent across dataset splits.
(d)  Table 2 is really hard to parse, lacking in description and in general confusing. As per my understanding, CSRS.R.TV.VGG vs the proposed framework should be compared across the same anatomical class. If so, the differences in effect sizes appear very minor (third decimal place) for several comparisons.

2. If my understanding is correct, the baseline comparisons in the paper correspond to evaluation against ablated versions of the framework (various loss functions), evaluation in the original resolution and linear interpolation. Given that the authors listed a couple of recent approaches proposed in literature in Section 1, it is not obvious why they did not include these as baseline comparisons.
questions: 1. Section 2.1 : "We then set weights relative to the value of the reconstruction error after convergence such that: the TV loss is roughly 2/3 the reconstruction term (R); the perceptual loss is roughly 3x R; the Dice loss is roughly equivalent to the perceptual loss. This strategy, based on our task-specific goals, enables us  to compare models consistently and add/subtract terms without extensive weight optimization."

How did the authors arrive at this heuristic?

2. Formatting issues

(a) Oddly large spacing after section 2 heading 

(b) The plots in the paper are a bit blurry and tables are included within figures. A suggestion would be to replace these with better resolution images and use the default latex package for displaying the tabular data.

3. Discussion "we trained each model for 2 epochs or until convergence", does this imply that the maximum number of epochs used is 2?
limitations: The main limitations discussed in section 6 including computation and quality of annotations. Given that the framework is so computationally expensive that 1 epoch takes 12 hours seems to greatly limit the practical utility.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work developed a super resolution method for 3D neuroimaging and evaluate its performance in detecting brain changes due to neurodegenerative disease. It trained on 3D brain data to upsample both the raw intensity image and associated segmentation labels. The method was mainly based on the 2D deep back projection network (DBPN) [5], which was extended to three dimension and multiple output. The method was evaluated in downstream clinically relevant signal detection problem: quantifying cross-sectional and longitudinal
change across a set of phenotypically heterogeneous but related disorders that exhibit known and differentiable patterns of brain atrophy. 
strengths_and_weaknesses: Strength:
-- This work conducted the evaluation of super resolution in the downstream clinically relevant tasks, e.g., quantifying cross-sectional and longitudinal change across a set of phenotypically heterogeneous but related disorders.

Weakness:
-- This work overall has limited methodology contribution. 

-- The organization of this work is not clear and hard to follow for some sections.

-- This work fits more to a dedicated medical imaging conference or journal.

questions: See the detailed comments above.
limitations: No.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present an extension of a 2D perceptual super-resolution architecture extended to 3D within neuroimaging. The authors additionally extend the network to simultaneously produce up-sampled segmentation maps. The upsampled segmentation maps are evaluated against the ground truth, as well as against detection power across a variety of frontotemporal disorders. The effect of different loss functions used in training are also evaluated against prediction power. Whilst the super-resolution techniques improved standard evaluation metrics (PSNR & SSIM) when compared against the low resolution data, these metrics did not significantly improve when compared against models trained with the different loss functions used. Prediction power for frontotemporal disorders was improved in many regions when using the super-resolution models, however some regions suffered in prediction power when super-resolved.


strengths_and_weaknesses: Overall this paper is clear and concise, and forms a well constructed manuscript. The background information and methods are extensive, and are presented in logical, clearly defined, sections. The overall study is fairly in depth for the given research question, and presents a nice analysis of the downstream effect of segmentation super-resolution within a disease model. The authors include the relevant previous work done in this area, and appropriately outline their contribution to this field.

The biggest limitation of this work I feel is the limitation of impact in some of novel contributions outlined. Specifically, extending a model from 2D to 3D, whilst relevant within neuroimaging data, is not particularly impactful. Additionally, adding segmentation maps as an additional output channel does improve user convenience, and likely a computational efficiency gain, but is ultimately not overly impactful as the segmentation maps could be re-calculated with the inferred high-resolution dataset. Overall however, it is my opinion that the strengths do clearly outweigh the weaknesses in this work.
questions: I have no questions at this point.
limitations: The authors have addressed the limitations of their work in the conclusions section.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a 2x (from 2mm to 1mm) super-resolution method for 3D medical image volumes and the corresponding segmentation mask for better brain diagnosis. The proposed method is a natural extension of the 2D super-resolution model called deep back-projection network (DBPN). Compared with linear upsampling and nearest-neighbor upsampling, the proposed method achieved a better dice score as well as a better visual quality. 
strengths_and_weaknesses: Strength:
This paper provides good implementation/technical details to illustrate the problem, the method, and the experiments. It also provides a good visual-based comparison to interpret the method's performance. It is interesting to consider upsampling both the image along with the segmentation mask.


Weakness: 
The method proposed in the paper doesn't seem to be novel given the fact that it is a natural extension from 2D to 3D based on the deep back-projection network (DBPN). Though the paper proposed a few loss functions to adjust the model's objective, the performance does not show a significant difference between different combinations of loss functions. 

Moreover, the paper doesn't provide a good description of the related work and the main comparison is done between the variants of the proposed method and the trivial methods (linear/nearest-neighbor upsampling).
questions: Is this the first super-resolution 3D model in this area? If not, can you make a comparison between the proposed method and the existing methods?
limitations: As stated in the weakness, the method is not novel to me and the experimental comparison is not satisfactory for a NeuroIPS paper. The paper lacks a thorough description of related work and a better comparison with other existing methods other than linear/nearest-neighbor upsampling.
review_title: null

Meta-review: metareview: This paper has mixed evaluations, with two reviewers recommending accept and three recommending reject. After carefully reading the paper and the discussion, I agree with reviewers hYgi, Ho1b, aKcw, Uzm9 in their main criticisms. The paper still requires major revisions before it can be accepted, including, but not limited to, an improvement in the clarity of the presentation and more experimental comparisons against other, perhaps, even simpler approaches.
confidence: Certain
Decision: Reject