Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a self-supervised deep-learning framework for image-to-image translation tasks, such as segmentation, that accommodates and fully exploits longitudinal data.  Specifically the method provides a mechanism to impose consistency in output across multiple points from the same individual and simple regularisation terms to avoid common problems with other methods, such as mode collapse. The authors compare the method against baselines in two distinct neuroimaging segmentation tasks, which nicely demonstrate the additional power afforded by imposing longitudinal consistency.
strengths_and_weaknesses: Strengths

- The work addresses a persistent challenge and shows a significant advance.

- The experimental work is well thought out and showcases nicely the potential of the idea.

Weaknesses

- (minor) In places the writing could be made more accessible to the non-specialist and organised a bit better. Highlighting specifically what the reader is looking for in Figure 1 would help interpret that figure.  The task isn't specified in the OASIS paragraph (starting line 202). These things can be figured out but the authors could make the reader's job easier.
questions: Broadly happy with it apart from the minor issues raised above.
limitations: Seems fine.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles self-supervised learning for longitudinal medical images by designing loss functions that consider the temporal evolutions of scans taken from the same subject with applications in downstream segmentation. A U-net architecture is pre-trained via a combined loss function that (i) maximizes the similarity between encoder features from corresponding local patches acquired at different times from the same subject, (ii) minimizes the similarity between encoder and decoder features at the same resolution, as well as at different channels to combat mode collapse and improve generalization, (iii) minimizes the reconstruction error between different augmentations of the same scans. Following pre-training, one-shot segmentation fine-tuning is augmented again with temporal consistency by maximizing the Dice score between segmentation predictions at different times from the same subject. Resulting method outperforms both randomly initialized U-Net, as well as several state-of-the-art pre-training methods with and without benefiting temporal information. 
strengths_and_weaknesses: Strengths: Self-supervised learning from longitudinal medical imaging is an important problem due to costly annotations. The paper is well-written, particularly in experiment setup. Generalization of state-of-the-art autoencoder pre-training principles to temporal consistency is novel.

Weaknesses: Motivation part of the introduction could be expanded and improved with some citations. 

questions: Following works also tackle self-supervised learning in longitudinal brain imaging. Can the authors address their differences with the current work?

Wei et al. “Consistent Segmentation of Longitudinal Brain MR Images with Spatio-Temporal Constrained Networks”, MICCAI, 2021

To et al. “Self-supervised Lesion Change Detection and Localisation in Longitudinal Multiple Sclerosis Brain Imaging”, MICCAI, 2021

limitations: Limitations and potential impacts have been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper claims that prior works on self-supervised learning are ill-suited to handle typical longitudinal medical image sequences with problems due to assumptions of i.i.d data and methods that are not able to do per pixel-level predictions. As such the authors propose a number of solutions:

1. a spatiotemporal patchwise similarity loss, which attempts to increase longitudinal similarity between features at similar spatial positions and at all layers of the network
2. enforcing orthogonality of features from the encoder and decoder.
3. encouring variance and decorrelation of decode features
4. denoising reconstruction loss
5. a finetuning term that encourages segmentations to be similar over multiple time points.

The proposed solutions are evaluated on three datasets with longitudinal medical images, in comparison with
a number of previously developed self-supervision methods aswell as standard supervised methods such as UNet
.

strengths_and_weaknesses: + Self-supervision in medical imaging and particularly in longitudinal analysis is very relevant and an impo
rtant topic, that in my opinion is not well addressed by existing solutions.

- Clarity, I find the work hard to follow with long sentences that often appear ambigious to me.    

- Quality and significance, what is the relevance and significance of the results? Variance over subjects and multiple training results is not presented. What is the relevance outside one-shot-segmentation?

- Soundness, I am not sure the observations and conclusions of mode collapse (Fig.1) is correct. There is little evidence presented. The speculation that this is due to skip connections between the encoder and decoder seems hard to verify from the presented evidence. This speculation is repeated multiple places and used as a design argument in sections "Methods - Orthogonality, Variance and covariance".
questions: - I have problems appreciating the proposed solutions as I do not understand and not convinced of the problem the authors are trying to address. Specifically the claims of mode collapse, supposedly illustrated in Figure 1. Could the authors do more to describe, quantify and illustrate this problem?
limitations: - Limitations are reasonably well addressed.
review_title: null

Meta-review: metareview: The paper proposes a self-supervised deep-learning framework for image-to-image translation tasks, such as segmentation, that accommodates and fully exploits longitudinal data. Specifically the method provides a mechanism to impose consistency in output across multiple points from the same individual and simple regularisation terms to avoid some problems common with other methods, such as mode collapse. The authors compare the method against baselines in two distinct neuroimaging segmentation tasks, which nicely demonstrate the additional power afforded by imposing longitudinal consistency.

The reviews overall reported that the submission tackles an important problem, presents well formulated experiments, and shows a significant advance over the SOTA. 

The reviews raised concerns raised included non-specialist accessibility, adding more related work, and questions w.r.t. the claims, presentation, and relevance of the results, and particularly about the mode collapse problem. The authors have addressed these concerns, in particular by adding a new section (Section E/Need for regularization) to the Supplementary Material which contains several new visualizations and quantifications of the mode collapse problem (from ablation experiments).
The paper has been updated to reflect some  clarifications required by reviewer de4n. Some citations suggested by Reviewer r9Zh are now discussed in the submission.

As it is, the paper meets all conditions for acceptance at NeurIPS 2022.

confidence: Certain
Decision: Accept