Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis
Authors: Mengwei Ren
Neel Dey
Martin Andreas Styner
Kelly Botteron
Guido Gerig
Venue: NeurIPS 2022 Accept (2022)
Keywords: neuroimaging
medical image analysis
spatiotemporal representation learning
longitudinal learning
medical image segmentation

Abstract: Recent self-supervised advances in medical computer vision exploit the global and local anatomical self-similarity for pretraining prior to downstream tasks such as segmentation. However, current methods assume i.i.d. image acquisition, which is invalid in clinical study designs where follow-up longitudinal scans track subject-specific temporal changes. Further, existing self-supervised methods for medically-relevant image-to-image architectures exploit only spatial or temporal self-similarity and do so via a loss applied only at a single image-scale, with naive multi-scale spatiotemporal extensions collapsing to degenerate solutions. To these ends, this paper makes two contributions: (1) It presents a local and multi-scale spatiotemporal representation learning method for image-to-image architectures trained on longitudinal images. It exploits the spatiotemporal self-similarity of learned multi-scale intra-subject image features for pretraining and develops several feature-wise regularizations that avoid degenerate representations; (2) During finetuning, it proposes a surprisingly simple self-supervised segmentation consistency regularization to exploit intra-subject correlation. Benchmarked across various segmentation tasks, the proposed framework outperforms both well-tuned randomly-initialized baselines and current self-supervised techniques designed for both i.i.d. and longitudinal datasets. These improvements are demonstrated across both longitudinal neurodegenerative adult MRI and developing infant brain MRI and yield both higher performance and longitudinal consistency.

Local Spatiotemporal Representation Learning for Longitudinally-conistted Neuroimage Analysis
Mengwei Ren New York University mengwei... ren@nyu. edu
Neel Dey New York University neel . dey@nyu. edu
Martin A. Styner
UNC-Chapel Hill stynerêcs. unc edu
Kelly N. Botteron
Guido Gerig
WUSTL School of Medicine
New York University
botteronk@@wsttl.co edu
gerig edu
Abstract
Recent self-supervised advances in medical computer vision exploit the global and local anatomical self-similarity for pretraining prior to downstream tasks such as segmentation. However, current methods assume i.i.d. image acquisition, which is invalid in clinical study designs where follow-up longitudinal scans track subject-specific temporal changes. Further, existing self-supervised methods for redicallly-reevant image-to-image architectures exploit only spatial or temporal self-similarity and do via a loss applied only at a single image-scale, with naive multi-scale spatiotemporal extensions collapsing degenerate solutions. To these ends, this paper makes two contributions: (1) presents local and multi-scale spatiotemporal representation learning method for image-to-image architectures trained on longitudinal images. It exploits the spatiotemporal self-similarity of learned multi-scale intra-subject image features for pretraining and develops eral fcature-wise regularizations that avoid degenerate representations; (2) During finetuming, it proposes surprisingly simple self-supervised segmentation con- sistency regularization to exploit intra-subject correlation. Benchmarked across various segmentation tasks, the proposed framework outperforms both well-tuned candomly-initiaiizz baselines and current self-supervised techniques designed for both i.i.d. and longitudinal datasets. These improvements are demonstrated across both longitudinal neurodegenerative adult MRI and developing infant brain MRI and yield both higher performance and longitudinal consistency.
Introduction
Tracking subject-specifp anatomical trends over time is crucial to both clinical diagnostics and large-scale biomedical science. Such longitudinal imaging is especially relevant to analyzing neuro- logical patterns of growth and degeneration via brain imaging in pediatric and elderly populations, respectively. As tracking individual structural changes requires precise and longitudinally-coniieeee segmentation methods with scarce annotated training volumes, identify two major bottlenecks in existing seff-supervised biomedical image analysis methods which we use to motivate our work.
Learning with few annotations. While modern imaging studies may hundreds to thousands of individuals, manually outlining volumetric structures of interest across multiple individuals for supervised segmentation training is prohibitively expensive. Therefore, current work focuses leveraging large sets of unlabeled images to pretrain mage-to-imaaee architectures (e.g., the U- Net [48]), which can then be efficiently finetuned in the one few-shot setting. These self-supe methods may handcraft pre-text training objectives [10, 19, 45, 65] or may attempt to pretrain the base
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Infant 24 mo.
U-Net Encoder
U-Net Decoder
untuperviled negittial
1/16 res
1/8 res
1/4 res
1/2 re5
neghtves
mo.
w/o negatives (ours)
Adult 81 yrs
oe negatives (ours negetives A. unsuperriged
regetives
75 yrs
Figure I: On pretraining an image-to-image network with per-layer spatiotemporal self-supervision, visualize the intra-subject multi-scale feature similarity between query channel wwse feature and all spatial positions within the key feature at different age. A: Contrastive pretraining with unsupervised negatives [44] yields only positionally deeennnddlddeeeeenttteeeee representations. B: Pretraining w/o negatives [11] by using corresponding intra-subject patch locations as positives leads to semantically- implausible representations with low-diversity (e.g., see yellow box) and artifacts (see arrows) in deeper layers. C: Our method attains both positionally and anatomically-relevan representations via proper regularization (e.g., see green box). Additional structures are visualized in Suppl. Figure 5.
network to be equivariant to transformationsin order to preserve semantic meaning as in contrastive learning in medical [9, 63] and natural image vision [2, 25, 38, 60, 67, 68). However, these methods typically leverage label supervision in sampling for their losses and impose a self-supervised loss only at single feature scale (typically the encoder bottleneck network output). Naive application of local unsupervised multi-scale contrastive losses [16, 44] lead to degenerate representations (Fig. row A) when applied both the encoder and decoder of image-to-image architectures.
Violating i.i.d. assumptions. Further, most existing self-supervised frameworks assume i.i.d. data. Unfortunately, this assumption does not transfer to longitudinal studies where intra-subject temporal images are highly correlated. Emerging longitudinal representation learning methods focus on impos- ing camporal-consistenn
into the encoder bottleneck [15, 43, 66], such that the encoder learns repre- sentations that are aware of the order of acquisition [15] and the overall trajectory [43]. These methods address image-level tasks such as disease classification or progression and age prediction. However, their extension to pixel-level applications with image-to-image architectures remains unclear.
Methods. Motivated by the above limitations, in this work, claim that the spatiotemporal de- pendency of imaging data should be explicitly incorporated in self-supervised frameworks. We do so by exploiting the spatial and temporal self-similarity of local multiscale deep features in both encoder and decoder and further learn diverse intermediate representations by developing regularizations for self-supervised similarity objectives. Lastly, when finetuning with limited annotated data, encourage predictions on unlabeled subject-wise images to be spatiotemporally consistent.
Contributions. This work makes the following contributions: (1) It presents longitudinally- consistent spatiotemporal representation learning framework to learn from image time-series; (2) To impose multi-scale local self-supervision while avoiding degenerate solutions, develops regulariza-

---

tion terms on the variance, covariance, orthogonality local features within the decoder; (3) To further self-supervise the fine-tuning stage, the proposed method encourages segmentations from adjacent timepoints on unlabeled data to be consistent; (4) Across three large-scale longitudinal one, few, and full-shot segmentation tasks on both elderly and pediatric populations, the developed framework yields improved segmentation performance and higher longitudinal segmentation consistency. Our code is available at https: //g /gtgtutt.bb.. con/mengxeirren/ongg tudinal saarreesennaiio-neeennieeee
2 Related work
Self-supervision. Self-supervised learning (SSL) methods aim to learn hierarchical representations from unannotated data, which can then be transferred to tasks operating in low-annotation regimes. Early work focused pretext tasks where handcrafted loss is used to pretrain networks via orienta- tion prediction [19], context restoration [10, 45], channel prediction [65], among others. Given their heuristic nature and suboptimal generalization, recent work instead focuses on data-driven SSL losses.
Contrastive learning. Constrastive learning [11, 22, 23, 24, 49, 55] (CL) typically transforms an input image and asks the embeddings of the input image and its transformation (the positive pair) to be close to one another and far apart from embeddings of other images (negatives) via a noise contrastive estimation [21, 42] (NCE) loss. While performant on image-level recognition tasks [12, 32]. CL requires non-trivial modification to extend to pixel-level segmentation tasks, as described below.
Negative-free representation learning. In several applications, true negative samples may be diffi- cult construct [26]. For example, when learning on intra-domain internal image patches [16, 44], non-local spatial positions may be semantically similar, but NCE objectives push their embeddings apart, leading to false negative pairs introducing label noise in the training objective. This draw- back may be mitigated via data-driven SSL methods which only use positive samples and avoid low-diversity (or collapsed [30]) embeddings solutions via predictor networks and custom backpro- pogation [13, 20] and careful regularization [6, 62]. However, above, these methods operate on global image embeddings and require modification for pixel-level tasks.
Spatial self-supervision. Towards downstream segmentation, recent work [2, 35, 38, 60, 67, 68] encourages local single-scale features either within an image or across images to cluster semantically by constructing positive pairs using ground-truth labels. To incorporate local and multi-scale spatial considerations into unpaired image translation and registration [16, 44] imposed contrastive losses on randomly-soo sssmple layer-wise wise encoder features by considering corresponding spatial indices as positives and all other locations as negatives. Our work builds on this by instead only considering temporal positives (described below) in the layerwise losses alongside custom regularization which avoids low-diversity decoder embeddings observed with naive application Fig. 1.
Temporal self-supervision. SSL methods developed for video achieve high performance by exploit- ing temporally consistent transformation [5] and temporal pretext task [14, 29, 59]. However, longitu- dinal biomedical image time-series have sparser sampling (typically 2-5 timepoints/subjecc) and have greater spatial extents (volumes instead of images), which leads distinct modeling considerations.
Emerging biomedical methods [15, 43, 66] enforce smooth trajectories for subject-wise images in the encoder latent space and deploy their methods on image-level downstream tasks such as disease classification and age regression. However, these methods focus on learning a global embedding, without clear extension to pixel-level tasks such segmentation.
Biomedical image segmentation. Major challenges specific to biomedical segmentation include: (1) large 3D volumes; (2) limited sample sizes and annotations; and (3) non-i.i.d. longitudinal acquisitions tracking temporal anatomical changes. To these ends, conventional approaches use a combination of intensity-based probabilistic models and registration-driven atlas-based models [1, 27, 39, 58]. In particular, longitudinal image analysis typically makes use of one or few longitudinal atlases [28, 33, 47, 51, 52], which motivates the one and seo-shot segmentation settings benchmarked in this paper, respectively.
More recently. deep segmentation networks achieve strong performance [8, 7, 40, 41, 48] given enough training volumes. In the low-annotation setting, weakly supervised methods develop custom loss functions [31, 37], but may have drawbracks analogous to the handcrafted SSL losses described above. Fortunately, recent data-driven self and semi-supervised methods are well-suited to pixel-level prediction. For example, to pretrain an encoder for segmentation, [9, 63] develop
3

---

Pretraining
Finetuning
6 miggith
12 g
24 month
age
ago 6 mmonth
36 month
Net Feature space
Puy
Latent feature space
U.SeT
Decoder feature space
uomporsll mogtts
U Not
Pish oper
jul
Temporal patch similarity loss
(b) Orthogonality loss
(c) Spatial variance
(d) Covariance
Consistency loss
Figure 2: Overview of proposed self-supervision. Given nontinearly-registeeee temporal images of a subject, (a) we assume that corresponding spatial locations in various network layers should have similar representations. As U-Net skip connections can cause degenerate decoder embeddings (see App E), we (b) encourage the decoder bottleneck to be orthogonal to encoder bottleneck and regularize the concatenated decoder features to have (c) high spatial variance and be (d) uncorrelated channel- wise. During fine-tuning, we (e) encourage temporal intra-subject network output consistency.
application-speeee speccffc positive and negative sampling strategies for contrastive training, where 2D slices from similar locations in registered 3D volumes across subjects constitute positive pairs. While these methods have been successful in their applications, they are inherently slice-based methods and are outperformed by well-tuned randomly-inittalized 3D baselines our datasets (Tab. 1). Further, these self-supervised biomedical segmentation methods do not explicitly account for non i.i.d acquisitions. Lastly, to our knowledge, existing longitudinal deep learning work developed for biomedical segmentation is currently very specific to its target application [18, 36, 56] (for example, in tasks such MS lesion change detection [56]) or require supervised pre- pre-training on annotated cross-sectional datasets [61], whereas we develop a generic self-supervised spatiotemporal representation learning framework for non-i..d. longitudinal data which can be applied to any downstream task in principle.
3 Methodology
Fig. 2 illustrates the proposed framework. The base U-Net architecture is pretrained end-to-end in self-supervised manner with intra-subject spatiotemporal losses. On convergence, the pretrained network parameters serve as an initialization for training downstream local spatiotemporal pixel-level tasks (e.g., registration segmentation). This work will focus downstream segmentation in the one, few, and full-shot regimes. The main similarity loss is applied on multiscale local patches from different timepoints of the same subject, which attracts features in corresponding locations in separate timepoints together. The high-dimensional U-Net features corresponding to the boxes at the same locations in Fig. 2a should maintain high similarity despite varying appearance. The feature regularizers (Fig. 2b,c,d) avoid degenerate U-Net decoder embeddings in the patch similarity training and the output regularizer (Fig. 2c) encourages finetuning consistency on unannotated data.
Setup. The unlabeled dataset is a collection of N subjects, where each subject has at least two longitudinal image acquisitions available during pretraining. During every iteration, pair of images (X)22)+) 21+1) are randomly sampled from subject E 21,2,.... .......N...... } at distinct timepoints j and j + 1, where j E 21.2..... , T, - 1}. T, indicates the number of registered images from subject i, and X' E RW*H xxxDXC is 3D volume of spatial dimension W H x D and C' channels. These channels are typically multi-modality acquisitions (e.g., T1w and T2w MRI from the same subject).
Spatiotemporal patchwise similarity loss. We aim to associate the local embeddings of X3 and x14 such that the representations are longitudinally-aware. To this end, weight-sharing 3D U-Net G takes both images input and produce a set of multi-scale CNN features (V) where each element Uay = (xx)) RR,, HI DDxC; indicates the output of the Ith layer of interest. M feature vectors are then randomly sampled from the 3D spatial indices of the feature map, where each feature vector uft represents local patch of the input image. These patch-wise activations from multiple layers of G' form hierarchical representations of local regions of the input image.

---

To maximize the similarity of corresponding local features (e.g., blue boxes in Fig. 2a) without negative samples, we use projector MLP f and predictor head p and extend [13] to patchwise operation such that matching spatial indices have high agreement. The patchwise similarity loss between two representations is defined as follows: [(vyy. m(kk) 2 PD(1, 22) + 2 DDP2, 21), where
D(p, =) Tllle Talla" 21, #2 = f(0jz) ), ((fjss and pt. P2 - p(x)), The total loss is an
P_
average of all sampled patches, across multilayer features:
Leim
L
1
Σ IE(L....L)
1
E
M
mE(1.....M)
Architectural challenges in multi-scale representation learning. Laim applied to the hidden layers of U-Net is found to maximize patchwise similarity of the encoder features (as expected) but lead to low-diversity and similarity maps in Fig. 1B and empirical observations in App. E. We speculate that this is partially attributable to U-Net skip connections, which lead to degenerate solution where the encoder learns representations which are good enough to minimize the decoder losses and the decoder layers do not have to learn useful representations which transfer. To this end, we develop several regularization strategies such that the decoder layers obtain diverse and wawaaaniiclll-coheeeee embeddings.
representations in the decoder layers as observed from the
Orthogonality. During model prototyping, we empirically observe that low-diversity embeddings first originate in the U-Net bottleneck (Fig. 1B cols 4,5 and App. Fig. 14 rows 2,3) which then upsampled hierarchically through the decoder. We therefore encourage decoupled bottleneck features between the encoder and decoder. Revisiting the U-Net skip-connection, the encoder features from the 1-th layer E RW H; XCi are concatenated with the upsampled decoder features followed by a convolution to attain the same feature dimension Us. This yields va = Conv Manv(Concat(u) (ve, va)) ) E R x, xCi and by regularizing for orthogonality between Ve and 2 in the projector embedding space, we implicitly encourage va to learn better representations instead of converging to degenerate solutions. The orthogonality loss is defined as
Va R W.
Lol 22)
(2)
M
mE(.....M)
where = and 2 are projected representations via f at sampling location m. from the matched encoder/decoder layers, respectively.
Variance and covariance. We further encourage spatial variation and channel decorrelation of local decoder features to avoid degenerate representations. We extend the variance and covariance terms of [6] towards patchwise multi-layed operation and apply them the decoder layers. A standard deviation loss encourages spatial feature variation above a threshold of r) = 1 and is defined as
Σ Ls(z) IE(I.-R,...L.
(max(0, 1 S(z));
where S(z) = /Vr(zz) + e is the standard deviation of M randomly-sampled projected features from the 1-th layer and E = 10-4 is added for numerical stability. A covariance regularization decor- relates channelwise activations in the decoder to prevent low-diversity embeddings by minimizing
Co(2) k
1
Σ
1 IIC(E)) n
tEIL-k....L
where -(=)=,-1 - Σ. =) 2) is the covariance matrix of n-D representation and (u, v)
indicates its off-diagonal indices.
Reconstruction. To further pretrain the decoder, we investigate reconstruction losses commonly used in unsupervised learning. However, as high-resolution information is passed through skip connections, reconstruction loss for U-Net is near-triviee -rivialiv minimized and does not address degenerate solutions. Therefore, inspired by denoising autoencoders [57], we encourage network equivariance and invariance geometric (AA,) and intensity -ased transformations (Ai), respectively, of the input image æ, by modifying the reconstruction objective to a denoising loss as,
Free [G(A,,A, - 11)
(5)
5

---

Input
Randinit. 30
Context
LINE
PCL
Ours
Reference
IBIS arggm
I81S SiS--subortt
OASIS3
ingut
Context
INE
Curs
IBI BBI5-wmom
Figure 3: One-shot segmentation. Top rows: Once pretrained on all unlabeled data, all bench- marked methods are finetuned on either single annotated image (IBIS-wmgm) or a single annotated subject (IBIS-subcori and OASIS3). When deployed other subjects at different ages, our method yields improved segmentation performance. Bottom row: When finetuned only on single 36 month-old image, method generalizes to unseen timepoints by leveraging temporal consistency.
Finetuning with longitudinal consistency regularization. longitudinal segmentation, the finetuning data do not cover the overall age range, the network may perform well on the finetuned timepoints, but may perform poorly on unseen ages even when pretrained on unannotated data across all ages. We therefore develop self-supervised longitudinal consistency regularization term applied at the network output during finetuning to increase the intra-subject agreement. Given registered and unannotated image volumes, we formulate segmentation prediction consistency loss as,
Les = Dice (G(x)), G(z)+1));
(6)
which is minimized alongside the supervised segmentation term below during finetuning.
Total objective. During pretraining, our overall objective function weighted sum of the patch similarity loss, reconstruction loss, and three regularizations, and is defined as,
LP XLsim + arCe + pCs (z) + 7Cc(=) (L)
During finetuning, we use combined dice and cross entropy loss supervised training pairs as,
Coup (1 Dice (Gle), v)) + CE(G(z).
where is the groundtruth segmentation label. Lastly, we additionally use the segmentation consis- tency loss Ces for non i.i.d inputs, which forms our final finetuning loss E Lsup + Las.
Experiments
Data and segmentation tasks. We conduct experiments two de-identified longitudinal neuroimaging datasets, and specifically design three tasks to benchmark different extents of biomedical domain gaps between the finetuning and testing data. The main body of this work focuses on one-shot segmentation using one annotated subject (a common medical image analysis setting). Benchmarks on few-shot and fulevssup segmentation tasks are provided in Appendix A. For both datasets, we perform vrain/validtiion/ess split on a subject-wise basis with 70%, 10% and 20% of the participants. The validation set is used for model and hyperparameter selection and results are reported on held-out test set. ANTs [3, 4] is used to perform inter-subject affine alignment for all experiments, followed by subra-subject deformable registration to obtain accurate spatiotemporal correspondence for Lxim and L.s calculation. All images are skull-stripped, bias-field corrected. and intensity-normalize.. Further data preprocessing and splitting details are described in Appendix C.
OASIS3 [34] is publicly-avallable dataset consisting of 1639 brain MRI scans of 992 longitudinally imaged subjects. Each subject has 1-5 temporal acquisitions over 5-year long observation

---

window, resulting in an aging cohort over the span 42 to 95 years which includes cognitively normal and mildly impaired individuals alongside subjects with Alzheimer's s Disease. On OASIS3, we tackle whole-brain segmentation using the FreeSurfer label convention [17]. Cross-sectional FreeSurfer anatomical segmentation was done as part of the data release. Observing strong temporal inconsistency (see App. C.2), we further perform longitudinal FreeSurfer [47] to improve the temporal consistency of the reference segmentation. We exclude labels that have less than 100 voxels in all subjects, which results in 33 labels for segmentation training and evaluation. Finetuning for one-shot segmentation is performed on a single FreeSurfer--nnnttted subject with four timepoints.
IBIS is an infant brain imaging study, which longitudinally acquires 1272 structural T1w/22w MRI from 552 infants across both controls and infants at a high-risk for Autism Spectrum Disorder (ASD) over a span of 3 to 36 months of age. We tackle two distinct tasks: subcortical segmentation (IBIS- subcort) and white/gray matter tissue segmentation (IBIS-wmgm). For /BIS-subcort, a multi-atlas method [54] pross-sectionally segments sub-cortical grey matter (relevant to ASD [49]) into 13 structures of interest, which are then followed by manual corrections. For one-shot benchmarking. finetuning is performed only single longitudinalll-labedd
subject, similar to OASIS3 above. Further, we use the IBIS-wmgm setting to simulate a real-world use-case detailed in App. D.2. Briefly, brain MRI segmentation into grey/white matter is straightforward at 24-36 months of age due to the presence of anatomical edges in the images. However, ~6-month-on grey/white matter brain segmentation remains elusive without manually labeled datasets for supervision due to white matter myelination leading to isointense appearance at that age [53] (c.g. Fig. 2a and App. D.2). We therefore investigate finetuning all benchmarked methods on single 36 month old image which can be reliably segmented with [46] and then evaluate segmentation deployment with a strong domain shift 6 month old isointense images and labels (whose ground truth labels are generated by a fully supervised external model [64]).
Baselines and Evaluation Strategies. We analyze segmentation performance and longitudinal consistency against well-tuned randomly initialized 2D/3D U-Nets (RandI tunet......tt...... 2D/3D) and various high-performing self-supervised pretraining methods. These include the pretext-task based Context Restoration [10]. longitudinal representation learning based LNE [43], along with the contrastive learning based GLCL [9] and PCL [63] methods which operate on image slices. We also repurpose PatchNCE [44] for segmentation to evaluate its generic representation learning capabilities. All methods are pretrained and finetuned with both geometric and intensity-based augmentation, and share the same network architecture.
We quantify network performance via commonly used scores such as the Dice coefficient, IoU, and the 95-th percentile of the Hausdorff distance. More importantly, we also quantify the longitudinal agreement between intra-subject non-linearly registered temporal segmentations via scores such as the 2|8, where S1 and S2 are temporal [Si|+|Sa|
spatiotemporal consistency of segmentation [36] STCS
segmentation predictions from non-linearly registered input images; and the absolute symmetrized per- cent change [47] PS = 100 from S and S2. We also report ST and PSPC on the groundtruth segmentations as a reference.
VV=Vil where Vi2 and V/2 are the volume of a structure calculated 0.5 VV)
Implementation details. We train 3D U-Net [48] the base image-to-image architecture with four levels of up/down sampling and repeated Conv-BN-ReL.. blocks (all architectural details are provided in App. B). The projector head consists of 3-layer MLP with 2048 nodes per layer. Following [13], we apply batch normalization after each MLP, followed by ReLU activation, and l2 normalization of the final activation. The predictor is l-yer bottlenecked MLP with widths of 2048-256-2048 We apply geometric (left-right flip, random affine warps) and intensity augmentations including random blur, noise, gamma contrast enhancement. Additional MRI-specific augmentations with random bias field and motion artifacts are also applied and are followed by 128 random spatial cropping. We use a batch size of 3 crops and an initial learning rate of x 10 for both pretraining and finetuning. All networks are trained with the Adam optimizer (3, = 0.9 during pretraining and 3, = 0.5 during finetuning and f32 = 0.999 in both settings) on a single Nvidia RTX8000 GPU (45GB vRAM). The networks are pretrained for maximum of 30, 000 steps and the best model based on validation performance is used for fine-tuning for another 35, 000 steps, alongside lincar learning rate decay. fixed random seed due to limited computational budgets. Based on
All experiments are run the ablation analysis in Tab. 2. we empirically choose À= = L, = 10, 7 = 1e-3, B = 100 for all datasets, and use μ 10-2 for OASIS3, μ 10-3 for IBIS. Further details on the configurations, implementation of our method and other baselines are provided in Appendix B.
7

---

Table One-shot segmentation benchmarking of performance (median IoU & HD95) and longitu- dinal consistency (ASPC). Further scores alongside means and std. dev. are provided in Supplemental Table Few-shot and fully-supervised results are provided Suppl. Tabs. and 4, respectively.
IBIS IBIS-subcort
IBIS-wmgn
OASIS3
Method
IoU f
HD99
ASPCI
loUt
HD954
ASPCL
IoU f HDD9
ASPCI
GT
7.819
3.947
RandInitUnet.2D
0.707
4.485
7.127
0.510
3.274
11.506
0.687
2.545
10.189
RandInitUnet..3
0.720
2.892
4.644
0.560
3.788
5,515
0.715
2.206
2.789
Context Restore [10]
0.711
4.403
7.831
0.444
8.273
29.235
0.717
3.323
5.577
LNE [43]
0.736
3.033
5.866
0.563
3.201
5.352
0.726
1.988
8.836
GLCT [9]
0.718
3,203
5.514
0.550
4.112
8.472
0.695
2.264
4,622
PCL [63]
0.713
3.270
5.610
0.562
4.974
10.648
0.707
2.327
4.850
PatchNCE [44]
0.743
1.266
5.780
0.607
4.344
3,782
0.738
2.275
3.114
Ours w/o Les
0.754
1.145
5,483
0.614
3.291
2.462
0.739
1.940
2.729
Ours w/ Les
0.757
1.178
4.475
0.676
3.237
4.155
0.737
2.094
2.754
IBIS-wmgm
OASIS3
Dice
839
849
836
.871 -850
.712
.716
.718
.006
.818
.805
.819
.673
.885
.707
.755
.814
.817
0.90
0.8
0.95
0.7
0.6
0.80
STCS
.977
.913
911
.110
.955
.886
882
.888
.81/
-913
.910
.921
.919
.928
0.95
0.93
0.96
0.90
0.90
0.85
0.85
0.64
(BIS-SubCort(GT)
OASIS3 (GT)
RandInitUnet.20
RandInitUnet.3
Ours w/ Lex Figure 4: One-shot segmentation benchmarking quantifying performance with the Dice coefficient (top) and the spatiotemporal consistency of segmentation (bottom), visualizing the means and standard deviations alongside median values overlaid on the top of each subfigure (higher is better). Few-shot and fully-supervised results are provided in Suppl. Tabs. and respectively.
Context Restore
LNE
GLCL
PCL PatchNCE
Segmentation and longitudinal consistency results. Fig. 3 qualitatively demonstrates improved generalization using our method unseen longitudinal data (row 1-3), especially data displaying rapid intra-subject temporal developments (IBIS-wmgm, suncorr.... These improvements are consistent with the quantitative results presented in Fig. 4/Tab. which indicate both improved segmentation performance and longitudinal consistency. In the strong domain shift setting of IBIS-wmgm, we see a near ten-point increase in median dice over most baselines. With moderate shifts in IBIS-subcort, see appreciable increases in performance and consistency. We note that brain segmentation on adult brain MRI data from OASIS3 is comparatively casier task as adult neuroimages do not significantly change appearance between imaging sessions. Therefore, several baselines are able to match (but exceed) the segmentation performance of our method on OASIS3. However, all baselines are outperformed by ours on all datasets in terms of longitudinal-conisst i.i.d. statistical analysis. In particular, both our pretraining (Durs w/o Lex) and finetuning (Ours w/ Les) methods show STCS and ASPC improvements over all of the compared settings. Fig. 3 (bottom row) shows an example temporal predictions on two unseen timepoints from IBIS-wmgm. The predictions from Context Restoration [10] match only the input image intensity and lack anatomical and longitudinal consistency, LNE [43] introduces false positive predictions in temporal lobe (within the orange circle), and the proposed method yields more spatiotemporally and anatomically
which is essential to non-

---

consistent segmentation. Beyond one-shot segmentation, we also observe gains in the few-shot and fully-sueervised segmentation settings in Appendix A and Suppl. Tabs. 3 and 4, respectively.
Qualitative self-supervised spatiotemporal similarity. In Fig. 1, we qualitatively examine the learned visual representations of the proposed method via intra-subject temporal self-similarity (C) and compare it to two of its variants which either use contrastive learning with unsupervised negatives (A) or negative-firee representation learning (B). We calculate the per-layer multiscale feature self- similarity between the query and each key from the intra-subject feature maps at a different age (blue box). In row A, we see that assuming that all spatial indices not in correspondence constitute negative pairs leads to meaning (e.g., in the adult data, the similarity to localize to the ventricles in the coronal view). By and low-diversity embeddings
dependent representations in the decoder which carry low semantic
discarding all negatives in row B, we observe semantically-incoheee and artifacts in the decoder layers on both datasets. Finally, with careful regularization in row C, our methods discards all negative pairs and attains semantically and positionally relevant representations.
Ablations. As the proposed method consists of several moving parts, an ablation analysis is conducted over different model configurations, hyperparameters, and loss functions, reported in Tab. consisting of average dice coefficients. The combination of all proposed components yields optimal results. Further ablations and baseline tuning results are reported in Appendix A.
Row A starts with a base setting where only four encoder layers from the U-Net are selected for fesm computation and a small MLP width of 256 is used for the projection and prediction heads. Here, IBIIS-subcort and OASIS3 results are competitive with randomly initialized U-Net, as expected given the lack of auxiliary losses, data augmentation, and regularizations. However, the IBIS- experiment already shows a 2% improvement over random initialization, indicating benefits of using patchwise similarity losses for better out-of -iit----istribttion generalization even with suboptimal setups.
Row B: With larger projector and predictor networks, we observe improvements on two out of three datasets, which is consistent with trends observed on natural images [12, 13].
Rows C-F: On adding decoder layers to Lsim and introducing Cree alongside data augmentation (without any regularization), we typically observe inconsistent dataset- speciffc trends which arise from unregularized representations (e.g., Fig. IB). We speculate that poorly trained decoder (due to a lack of regularization) may be equivalent to random initialization in the context of pretraining for segmentation tasks. However, a combination of these components (Row F) leads to an appreciable increase in performance.
Rows G-K: When orthogonal regularization and/or covariance/variance regularization is used, we ob- serve the best performance when they are applied together alongside augmentation and & In rows J and K, we observe that different hyperparameters are optimal for OASIS3 and IBIS (which already outperform all baseline methods in Tab. 1), which is intuitive as these are drastically different cohorts.
Row L: Finally, the overall proposed model is achieved when Lee is added to the finetuming objective, which yields strong improvements for IBIS- (BIS-(wmgge subcor)) and maintains OASIS3 performance.
5 Discussion
Limitations and future work. The presented work opens up many follow-up questions which will be tackled in future work: (1) Our proposed losses enable better training and performance, but require the tuning of several regularization weights and layer selections which may reveal dataset specific patterns (e.g., rows J, K in Tab. 2). The weights and layers selected here were chosen based on limited exploratory experiments on the validation sets due to computational budgets and future work will exhaustively search the hyperparameter space for optimal performance. (2) Our pretraining assumes accurate non-linear intra-subject registration, which may be non non-trivial in edge cases like modalities with strong distortion and artifacts (e.g., eddy corruption in diffusion MRI). However, when studying pre and post-operative imaging (e.g., surgical excision of lesions), large topological changes break the assumptions of our model and will require the development of ussion-makkd positive patch sampling methods. (3) We use a two-stage pre-training and finetuning approach and it is plausible that the proposed method can be reduced to single-stage combined framework. (4) While this paper focused on downstream segmentation finetuning, the pretraining framework is generic to any pixel-level task (e.g., registration) and its extension to such tasks will be explored. (5) While the proposed methods yield strong longitudinal segmentation consistency improvements across all datasets, note that

---

Table 2: Ablation analysis of our method over loss layers, projection--predicioon layer widths (HMLP), loss functions (Ress L,s), use of augmentation, and hyperparameters (3, μ. 7). Mean dice is used
for quantification on all datasets. *p 10 on IBIS- (wmgmm subcort) ) and μ = 10-2 on OASIS3.
Exp
Loss Layers
#MLP
Leee
Aug.
B
Les
IBIS-subcori IBIS-wmgo
OASIS3
A
Enc
256
0.829(0.068)
0.733(0.062)
0.783(0.16)
B
Enc
2048
0.849(0.060)
0.809(0.13) 0.732(0.073)
C
Enc
2048
0.859(0.058)
0.713(0.079)
0.811(0.13)
D
Enc
2048
0.860(0.058)
0.810(0.13 0.718(0.066)
E
EncDec
2048
0.858(0.060)
0.724(0.077)
0.809(0.
F
EncDec
2048
V
0.856(0.060)
0.739(0.067)
0.812(0.13)
G
EncDec
2048
100
0.857(0.062)
0.728(0.074)
0.809(0.13)
H
EncDec
2048
10 3
10
0.845(0.063)
0.739(0.074)
0.804(0.13)
I
EncDec
2048
V
100
10
10 10-3
0.8590.056)
0.735(0.061)
0.811(0.13)
J
EncDee
2048
100
10-3
10
0.863(0.057)
0.808(0.14) 0.758(0.062)
K
EncDec
2048
V
100
10
10 10-3
0.853(0.055)
0.745(0.058)
0.813(0.13)
L
EncDee
2048
100
10
0.870(0.052)
0.806(0.030)
0.810(0.13)
absolute segmentation performance gains in elderly cohort (OASIS3) modest in comparison to the rapidly developing infant dataset (IBIS) where higher gains are achieved. Future work will further investigate these performance differences between populations of differing temporal trends. (6) We extended the negative-free frree framework of [13] to patchwise operations for its relative simplicity and it is plausible that other negative-free similarity terms [6. 20, 62] may further improve results. (7) In our data preparation for pretraining, subject-wise image time-series were registered to a single time-point instead of a subject-specific template, which known to increase statistical bias [47].
Limited scope. The proposed method generically applies to medical image time-serics and we do not anticipate negative impacts beyond those that currently exist for segmentation methods. However, while our tasks have impacts on understanding real-world disease mechanisms, we cannot claim any further insight into differences between subpopulations, such analysis requires close collaboration with clinicians, neuroscientists, and biostatisticians, which beyond the scope of this work.
Conclusions.
This paper addressed several open questions regarding the self-supervised pretraining and finetuning of image-to-image architectures on longitudinal volumes using objective functions which exploit both intra-subject spatial and temporal self- self-similarity. It developed local negative sample-free framework that trains multiple multi-scale hidden layers of image-to-image architectures that then enabled improved downstream segmentation performance, all while achieving semantically- meaningful representations via careful regularization of the decoder activations. During finetuning, it similarly developed simple sesisttency-rgulaiizaitiii
objective which encourages longitudinal agreement between predictions on unlabeled data. When applied to large-scale neurodeveloping and neurodegerative longitudinal images, the proposed framework yielded improved segmentation performance and temporal consistency, both of which are crucial to statistical analyses of mechanisms of interest such Alzheimer's Disease (OASIS3) and Autism Spectrum Disorder (IBIS).
Acknowledgements
The authors are grateful to NIH R01-HD055741-12, R01-MHH113362-02SL IROIMHI 18362- 01. IR01HD0S8 125-01A1, R01MH122447. R01- HD0058555.4 US4-HD0791244 124, P50-HD103573, R01ES032294, and the NYS Center for Advanced Technology in Telecommunication (CATT).
Adult longitudinal Tlw MRI data were provided by OASIS-3:NIH P50 AG00561, P30 NS09857781, PO1 AG026276, PO1 AG003991, RO1 AG043434, ULI TR000448, RO1 EB009352. Longitudinal developing infant brain T1w/T2w MRI were provided by the Infant Brain Imaging Study (IBIS) Network, which is an NIH-funded Autism Centers of Excellence (ACE) project and consists of consortium of 10 universities in the U.S. and Canada.
10

---

References
[1] Paul Aljabar, Rolf Heckemann, Alexander Hammers, Joseph V Hajnal, and Daniel Rueckert. Multi-atlas based segmentation of brain images: atlas selection and its effect accuracy. Neuroimage, 46(3):726- 2009.
[2] Iñigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C. Murillo. Semi- supervised semantic segmentation with pixel-level contrastive learning from a class-wise mem- ory bank. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 8219-8228, October 2021. 2,
[3] Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffeo- morphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26-41, 2008, 6, 26
[4] Brian B Avants, Paul Yushkevich, John Pluta, David Minkoff, Mare Korczykowski. John Detre, and James Gee. The optimal template effect in hippocampus studies diseased populations. Neuroimage, (0(3)2457-22460 2010. 6, 26
[5] Yutong Bai, Haoqi Fan, Ishan Misra, Ganesh Venkatesh, Yongyi Lu, Yuyin Zhou, Qihang Yu, Vikas Chandra, and Alan Yuille. Can temporal information help with contrastive self-supervised learning? arXiv preprint arXiv:201 13066, 2020. 3
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varimnc-invarianc--ccvaaiine regular- ization for self-supervised learning. arXiv preprint arXiv:2105.04990: 2021. 3, 5, 10
[7] Benjamin Billot, Magdamo Colin, Sean E Arnold, Sudeshna Das, Juan Iglesias, et al. Robust segmentation of brain mri in the wild with hierarchical cnns and no retraining. arXiv preprint arXiv:2203.01969 2022.
[8] Benjamin Billot, Douglas N Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V Dalca, and Juan Eugenio Iglesias. Synthseg: Domain randomisation for segmentation of brain mri scans of any contrast and resolution. arXiv preprint arXiv:2107.09559 2021.
[9] Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33:1 13:12256-1255 2020. 2, 3, 7, 8, 19, 21, 25
[10] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Self-supe learning for medical image analysis using image context restoration. Medical image analysis, 58:101539, 2019. 1, 3, 7, 8, 19, 25,
[11] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 2,
[12] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi- supervisd learners. Advances in neural information processing systems, 33:22243-2222 2020. 3,9
[13] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758, June 2021. 3, 5, 7, 9, 10, 34
[14] Hyeon Cho, Tachoon Kim, Hyung Jin Chang, and Wonjun Hwang. Self-supervised spatio- temporal representation learning using variable playback speed prediction. arXiv preprint arXiv:2003.02692. 8, 2020. 3
[15] Raphaël Couronné, Paul Vernhet, and Stanley Durrleman. Longitudinal self-supervision to disentangle inter-patient variability from disease progression. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 231-241. Springer, 2021.
2, 3

---

[16] Neel Dey, Jo Schlemper, Seyed Sadegh Mohseni Salehi, Bo Zhou, Guido Gerig, and Michal Sofka. Contrareg: Contrastive learning of multi-modality unsupervised deformable image registration. In Medical Image Computing and Computer Assisted Intervention MICCAI 2022.
Springer, 2022. 2,3
[17] Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Hasel- grove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness, et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3):341-355) 2002.
[18] Yang Gao, Jeff M Phillips, Yan Zheng, Renqiang Min, Thomas Fletcher, and Guido Gerig. Fully convolutional structured Istm networks for joint 4d medical image segmentation. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 1104-1108. IEEE, 2018. 4
[19] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803.078, 2018. 1,3
[20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supe-suue learning, Advances in Neural Information Processing Systems, 13:21271-212844 2020. 3. 10
[21] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirleenth international con- ference on artificial intelligence and statistics, pages -97-304. JMLR Workshop and Conference Proceedings, 2010. 3
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9733. 2020.
[23] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Interna- lional Conference on Machine Learning, pages 4182-4192. PMLR, 2020.
[24] R Devon Hjelm, Alex Fedorov, Samuel Larohe-Mar Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.08 06670, 2018. 3
[25] Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11291-16301. October 2021. 2,
[26] Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision, pages 2785-2795, 2022.
[27] Juan Eugenio Iglesias and Mert R Sabuncu. Multi-atlas segmentation of biomedical images: survey, Medical image analysis, 24(1):205-219, 2015. 3
[28] Juan Eugenio Iglesias, Koen Van Leemput, Jean Augustinack, Ricardo Insausti, Bruce Fischl, Martin Reuter, Alzheimer's ' Disease Neuroimaging Initiative, et al. Bayesian longitudinal seg- mentation of hippocampal substructures in brain mri using subject-specific atlases. Neuroimage, 141:542-555, 2016. 3
(29) Simon Jenni and Hailin Jin. Time-equivariant contrastive video representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9970-9980, October 2021.
[30] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. International Conference on Learning Representa- tions, 2022.
12

---

[31] Hoel Kervadec, Jose Dolz, Meng Tang, Eric Granger, Yuri Boykov, and Ismail Ben Ayed. Constrained-cne losses for weakly supervised segmentation. Medical image analysis, 54:88-99, 2019. 3
[32] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 1861-186773 2020. 3, 25
[33] Sun Hyung Kim, Vladimir S Fonov, Cheryl Dietrich, Clement Vachet, Heather C Hazlett, Rachel G Smith. Michael M Graves, Joseph Piven, John H Gilmore, Stephen R Dager, al. Adaptive prior probability and spatial temporal intensity change estimation for segmentation of the one-year-old human brain. Journal of neuroscience methods, 212((): 2013.
[34] Pamela LaMontagne, Tammie LS Benzinger, John C Morris. Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, et al. Oasis-3; longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv, 2019. 6, 16, 29
[35] Sangwoo Lee, Yejin Lee, Geongyu Lee, and Sangheum Hwang. Supervised contrastive embed- ding for medical image segmentation. IEEE Access, 9: 9:138403-138414 2021.
[36] Bo Li, Wiro J Niessen, Stefan Klein, Marius de Groot, M Arfan Ikram, Meike W Vernooij, and Esther Bron. Longitudinal diffusion mri analysis using segis-net: a single-step deep-learning framework for simultaneous segmentation and registration. Neurolmage, 235:1 18004, 2021. 4, 18, 19, 26, 27
[37] Shijie Li, Neel Dey, Katharina Bermond, Leon Von Der Emde, Christine A Curcio, Thomas Ach, and Guido Gerig. Point-supervised segmentation of microscopy images and volumes via objectness regularization. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1558-1562. IEEE, 2021.
[38] Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew Davison. Bootstrapping semantic segmentation with regional contrast. In International Conference on Learning Representations, 2022. 2,
[39] Jyrki MP Lötjönen, Robin Wolz, Juha R Koikkalainen, Lennart Thurfjell, Gunhild Waldemar, Hilkka Soininen, Daniel Rueckert, Alzheimer's Disease Neuroimaging Initiative, et al. Fast and robust multi-atlas segmentation of brain magnetic resonance images. Neuroimage, 49(3):2352- 2365, 2010.
[40] Fausto Milletari, Nassir Navab, and Seyed- -hhmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. IEFE, 2016. 3
[41] Andriy Myronenko. 3d mri brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop, pages 311-320. Springer, 2018. 3
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 07780000037... 2018. 3
[43] Jiahong Ouyang, Qingyu Zhao, Ehsan Adeli, Edith Sullivan, Adolf Pfefferbaum, Greg Zaharchuk, and Kilian M Pohl. Self-supervised longitudinal neighbourhood embedding. In International Conference on Medical Image Computing and Computer-Assiste Intervention. pages 89.89.06000000 Springer, 2021. 2, 3, 7, 8. 19, 21, 24
[44] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, pages 319-345. Springer, 2020. 2, 3, 7, 8, 19, 21, 23. 25
[45] Deepak Pathak, Philipp Krahenbuhi, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016. 3,
13

---

[46] Oula Puonti, Juan Eugenio Iglesias, and Koen Van Leemput. Fast and sequence-adaptive whole-brain segmentation using parametric bayesian modeling. Neurolmage, 143:235-249, 2016. 7, 29
[47] Martin Reuter, Nicholas J Schmansky, H Diana Rosas, and Bruce Fischl. Within-subject template estimation for unbiased longitudinal image analysis. Neuroimage, 6004)1140-, 2012. 3, 7, 10, 19, 26
[48] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisled intervention, pages 234-2411 Springer, 2015, 1,
[49] Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. arXiv preprint
2022. 3,7
[50] Mark D Shen, Meghan R Swanson, Jason J Wolff, Jed T Elison, Jessica B Girault, Sun Hyung Kim, Rachel G Smith, Michael M Graves, Leigh Anne H Weisenfeld, Lisa Flake, et al. Sub- cortical brain development in autism and fragile syndrome: evidence for dynamic, age-and disorder-specifie trajectories in infancy. American Journal of Psychiatry, pages appi-ajp, 2022.
16, 26
[51] Feng Shi, Yong Fan, Songyuan Tang. John H Gilmore, Weili Lin, and Dinggang Shen. Neonatal brain image segmentation in longitudinal mri studies. Neuroimage, 49(1) (9(1)391-400, 2010.
[52] Feng Shi, Pew-Thian Yap, Guorong Wu, Hongjun Jia, John Gilmore, Weili Lin, and Dinggang Shen. Infant brain atlases from neonates to l-and 2-year- PloS one, 6(4)xe18746, 2011.
[53] Yue Sun, Kun Gao, Zhengwang Wu, Guannan Li, Xiaopeng Zong, Zhihao Lei, Ying Wei, Jun Ma, Xiaoping Yang, Xue Feng, et al. Multi-site infant brain segmentation algorithms: The iseg-2019 challenge. IEEE Transactions on Medical Imaging, 40(5):1363-3)) 1376, 2021. 7, 29
[54] Meghan R Swanson, Mark D Shen, Jason J Wolff, Jed T Elison, Robert W Emerson, Martin A Styner, Heather C Hazlett, Kinh Truong, Linda R Watson, Sarah Paterson, et al. Subcortical brain and behavior phenotypes differentiate infants with autism versus language delay. Biological Psychiatry:
Cognitive Neuroscience and Neuroimaging, 2(8):664-672, 2017.
[55] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020.
[56] Minh-Son To, Ian G Sarno, Chee Chong, Mark Jenkinson, and Gustavo Cameiro. Self- supervised lesion change detection and localisation in longitudinal multiple sclerosis brain imaging. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 670-680. Springer, 2021. 4
[57] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and AntrreAntoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008. 5
[58] Hongzhi Wang. Jung W Suh, Sandhitsu R Das, John B Pluta, Caryne Craige, and Paul A Yushkevich. Multi-atlas segmentation with joint label fusion. IEEE transactions on pattern analysis and machine intelligence, 35(3);611 1-233,,,,,,, 2012.
[59] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Sele-supe video representation learning by pace prediction. In European conference on computer vision, pages 504-521 Springer, 2020.
[60] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Lue Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 7303-7313, October 2021. 2, 3
[61] Jie Wei, Feng Shi, Zhiming Cui, Yongsheng Pan, Yong Xia, and Dinggang Shen. Consistent segmentation of longitudinal brain mr images with spatio-temporal constrained networks. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 89-98. Springer, 2021. 4
14

---

[62] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self- supervised learning redundancy reduction. In Marina Meila and Tong Zhang. editors. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 12310-12320. PMLR, 18-24 Jul 2021. 10
[63] Dewen Zeng, Yawen Wu, Xinrong Hu, Xiaowei Xu, Haiyun Yuan, Meiping Huang, Jian Zhuang, Jingtong Hu, and Yiyu Shi. Positional contrastive learning for volumetric medical image segmentation. In International Conference Medical Image Computing and Computer- Assisted Intervention, pages 221-230. Springer, 2021. 2, 3, 7, 8, 19, 21,
[64] Guodong Zeng and Guoyan Zheng. Multi-stream 3d fen with multi-scale deep supervision for multi-modality isointense infant brain image segmentation. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 136-140. IEEE, 2018. 7,
[65] Richard Zhang, Phillip Isola, and Alexei A Efros. Split- Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1058-1067, 2017. 1,3
[66] Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Longitudinal correlation analysis for decoding multi--odal brain development. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 400-409 0000409...0 Springer, 2021. 2,
[67] Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10623-10633, October 2021. 2,
[68] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng. and Yu-Xiong Wang. Pixel contristive-consisseet semi-supervise supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7273-7282, October 2021. 2, 3
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract and Sec.
(b) Did you describe the limitations of your work? [Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you the full set of assumptions of all theoretical results? [N/A] No new theoretical results claimed.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material Code: code with instructions available at https://githrr com/mengweiren/ -angiittnnal---peeeeeeeeee
URL)? [Yes]
-earning. Data: public data links and proce- dures are described in the supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See 'Implementation details' in Sec. 4 and App. B.
(c) Did you report error bars (e.g., with respect ments multiple times)? [Yes] See Fig. 4, Tab. 1, Tab. 2.
random seed after running experi-
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See 'Implementation details' in Sec. 4.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See 'Baselines and Evaluation Strategies' in Sec. 4
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Our code is available freely
longi
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] See [34] and [50] for how imaging consent was acquired for OASIS3 and IBIS, respectively.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See 'Data and preprocessing' Sec. 4. Skull- stripping deidentifies human faces only the brain is available.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] See App. D.I.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] See App. D.
16


Reviews:
[{"review_id": "h3WXvetow6S", "reviewer": "Reviewer_r9Zh", "summary": "This paper tackles self-supervised learning for longitudinal medical images by designing loss functions that consider the temporal evolutions of scans taken from the same subject with applications in downstream segmentation. A U-net architecture is pre-trained via a combined loss function that (i) maximizes the similarity between encoder features from corresponding local patches acquired at different times from the same subject, (ii) minimizes the similarity between encoder and decoder features at the same resolution, as well as at different channels to combat mode collapse and improve generalization, (iii) minimizes the reconstruction error between different augmentations of the same scans. Following pre-training, one-shot segmentation fine-tuning is augmented again with temporal consistency by maximizing the Dice score between segmentation predictions at different times from the same subject. Resulting method outperforms both randomly initialized U-Net, as well as several state-of-the-art pre-training methods with and without benefiting temporal information. ", "questions": "Following works also tackle self-supervised learning in longitudinal brain imaging. Can the authors address their differences with the current work?\n\nWei et al. “Consistent Segmentation of Longitudinal Brain MR Images with Spatio-Temporal Constrained Networks”, MICCAI, 2021\n\nTo et al. “Self-supervised Lesion Change Detection and Localisation in Longitudinal Multiple Sclerosis Brain Imaging”, MICCAI, 2021\n", "limitations": "Limitations and potential impacts have been addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Self-supervised learning from longitudinal medical imaging is an important problem due to costly annotations. The paper is well-written, particularly in experiment setup. Generalization of state-of-the-art autoencoder pre-training principles to temporal consistency is novel.\n\nWeaknesses: Motivation part of the introduction could be expanded and improved with some citations. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OZQbYjWCg8r", "reviewer": "Reviewer_xiZS", "summary": "The paper proposes a self-supervised deep-learning framework for image-to-image translation tasks, such as segmentation, that accommodates and fully exploits longitudinal data.  Specifically the method provides a mechanism to impose consistency in output across multiple points from the same individual and simple regularisation terms to avoid common problems with other methods, such as mode collapse. The authors compare the method against baselines in two distinct neuroimaging segmentation tasks, which nicely demonstrate the additional power afforded by imposing longitudinal consistency.", "questions": "Broadly happy with it apart from the minor issues raised above.", "limitations": "Seems fine.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The work addresses a persistent challenge and shows a significant advance.\n\n- The experimental work is well thought out and showcases nicely the potential of the idea.\n\nWeaknesses\n\n- (minor) In places the writing could be made more accessible to the non-specialist and organised a bit better. Highlighting specifically what the reader is looking for in Figure 1 would help interpret that figure.  The task isn't specified in the OASIS paragraph (starting line 202). These things can be figured out but the authors could make the reader's job easier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1B5Dj2M7BsV", "reviewer": "Reviewer_de4N", "summary": "The paper claims that prior works on self-supervised learning are ill-suited to handle typical longitudinal medical image sequences with problems due to assumptions of i.i.d data and methods that are not able to do per pixel-level predictions. As such the authors propose a number of solutions:\n\n1. a spatiotemporal patchwise similarity loss, which attempts to increase longitudinal similarity between features at similar spatial positions and at all layers of the network\n2. enforcing orthogonality of features from the encoder and decoder.\n3. encouring variance and decorrelation of decode features\n4. denoising reconstruction loss\n5. a finetuning term that encourages segmentations to be similar over multiple time points.\n\nThe proposed solutions are evaluated on three datasets with longitudinal medical images, in comparison with\na number of previously developed self-supervision methods aswell as standard supervised methods such as UNet\n.\n", "questions": "- I have problems appreciating the proposed solutions as I do not understand and not convinced of the problem the authors are trying to address. Specifically the claims of mode collapse, supposedly illustrated in Figure 1. Could the authors do more to describe, quantify and illustrate this problem?", "limitations": "- Limitations are reasonably well addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Self-supervision in medical imaging and particularly in longitudinal analysis is very relevant and an impo\nrtant topic, that in my opinion is not well addressed by existing solutions.\n\n- Clarity, I find the work hard to follow with long sentences that often appear ambigious to me.    \n\n- Quality and significance, what is the relevance and significance of the results? Variance over subjects and multiple training results is not presented. What is the relevance outside one-shot-segmentation?\n\n- Soundness, I am not sure the observations and conclusions of mode collapse (Fig.1) is correct. There is little evidence presented. The speculation that this is due to skip connections between the encoder and decoder seems hard to verify from the presented evidence. This speculation is repeated multiple places and used as a design argument in sections \"Methods - Orthogonality, Variance and covariance\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept