# Self-Supervised Learning across the Spectrum
## 6 Experiments and Results
### Baselines  
We benchmark S4 against several competing self-supervised baselines. To the best of our knowledge, we are the first self-supervised approach that leverages multimodal imagery for SITS.  
**SatMAE:** SatMAE (2022) [32] is a SOTA Masked AutoEncoder-based vision transformer architecture. designed specifically for multitemporal satellite imagery. We implement SatMAE from the original codebase provided by the authors. Given that our work pertains to SITS, we use the multitemporal variant of SatMAE designed for SITS. We pretrain these models for 100 epochs. For segmentation, we similarly use a transpose convolutional neural network as a decoder that is trained during the fine-tuning stage. The authors of SatMAE also use a convolutional decoder when performing such experiments for downstream segmentation. We tuned the hyperparameters of the baseline to achieve the best possible performance on our two datasets during the evaluation.  
**SeCo, CaCO, and GSSL:** We compare against modern prior self-supervised work for remote sensing that uses single-modal contrastive loss. We compare against SeCo(2021) [28], CaCo (2023) [33], and GSSL (2021) [29], all of which use contrastive learning to align single-modal image pairs. In their original papers, these baselines use a single image ResNet-NN network to contrast scenes of different timestamps.  
We implement SeCo, CaCo, and GSSL from the original code base provided by the authors. All models use the MoCo-V2 architecture [67] with ResNet [68] backbone. We pretrain these models for 100 epochs. In the original implementation, a 2D UNet is used as the fine-tuned network. We found that the performance of this implementation was limited in our scenario since it is incapable of training on the entire SITS. Considering such limitations, our fine-tuning implementation for these baselines involves feeding each image in the SITS through the pretrained ResNet encoder and collecting the encoded feature maps of all of them. As a result, instead of one single feature map at each skip connection, we now obtain a time series of feature maps. We then pass the encoded image time series through a pixel-wise ConvLSTM decoder network to reduce the temporal dimension for fair comparison, before feeding the reduced single feature map into the upsampling part of the UNet-2D structure to achieve the final semantic segmentation predictions. We tuned the hyperparameters of the baselines to achieve the best possible performance on our two datasets during the evaluation.  
**Self-Supervised Multimodal Fusion:** This approach is a naive self-supervised approach of leveraging multimodal data for SITS segmentation. Let \(x_{m_{1}}^{l_{n}}\) denote the SITS modality we have access to at inference time. We first pretrain a network \(r(\cdot)\) that, given \(x_{m_{1}}^{l_{n}}\), learns to reconstruct the SITS of the other modality \(x_{m_{2}}^{l_{n}}\) (using the loss in Equation (3)). Then, we train a separate network that takes as input the concatenation of \(x_{m_{1}}^{l_{n}}\) and \(r(x_{m_{1}}^{l_{n}})\). Using this network, we produce the segmentation label using the PASTIS early fusion technique [43]. During inference, we similarly generate the SITS of the missing modality using the reconstruction network and perform segmentation on the original and generated modalities.