# Self-Supervised Learning across the Spectrum
## 5 Method  
Figure 2 provides an overview of S4. At a high level, S4 operates in three stages:  
* **Pretraining:** During the pretraining stage, S4 leverages abundant unlabeled data by jointly optimizing the proposed pixel-wise multimodal contrastive loss \(\mathcal{L}_{c}\) (Section 5.1) and reconstruction loss \(\mathcal{L}_{r}\) (Section 5.2): \[\mathcal{L}=\mathcal{L}_{c}+\lambda\cdot\mathcal{L}_{r},\] (1) where \(\lambda\) is a hyperparameter controlling the relative weight between the two loss terms. Neither of the above two losses require labels.
* **Downstream Training:** In this stage, our network is fine-tuned on the \(K\) SITS pairs with labels for downstream segmentation by further appending a segmentation module over the learned features (Section 5.3).
* **Inference:** In the final stage, S4 predicts a single segmentation map per different location from the SITS of a _single_ modality (either radar or optical).  
**Time Series Interpolation** A key challenge for S4 is that satellites visit the same location at different times, leading to temporal mismatch across modalities. Higher temporal mismatch across images causes more semantic mismatch in the underlying representation. To avoid this problem, we introduce a preprocessing strategy to coarsely align the temporal dimension between differing modalities. Our preprocessing strategy leverages **temporal metadata** from satellite imagery. This preprocessing step is necessary to ensure finer-grained spatial and time alignment through the rest of the training process. Recall that we are given as input \(x_{r}^{l_{n}}\in\mathbb{R}^{(T_{1}\times C_{1}\times H\times W)}\) and \(x_{o}^{l_{n}}\in\mathbb{R}^{(T_{2}\times C_{2}\times H\times W)}\), where \(T_{1}\
eq T_{2}\) in general. We determine which SITS modality has fewer time frames: let \(T_{min}=\min(T_{1},T_{2})\) and define \(x_{min}^{l_{n}}:=x_{r}^{l_{n}}\) if \(T_{min}=T_{1}\) and \(x_{min}^{l_{n}}:=x_{o}^{l_{n}}\) otherwise. The time series \(x_{min}^{l_{n}}\) remains unchanged. To make the other modality's SITS the same length, we adopt nearest-timestamp interpolation: for each image \(x_{min_{i}}^{l_{n}}\in x_{min}^{l_{n}}\), we find the image in the corresponding time series of the opposite modality that was captured at the time closest to \(x_{min_{i}}^{l_{n}}\). The result of our interpolation strategy results in \(N\) SITS pairs \(\{(x_{o}^{l_{n}},x_{r}^{l_{n}})\}_{n=1}^{N}\) where both modalities' time series each contain \(T_{min}\) images coarsely aligned in time.  
**Encoder Design** The first part of S4 consists of an encoder network that takes the spatially aligned optical and radar SITS, \(x_{o}^{l_{n}}\) and \(x_{r}^{l_{n}}\), as input. The encoder consists of four convolution layers, of which the first two are input-specific based on modality. Let \(y_{r},y_{o}\) denote the first two layers (used for the radar and optical domains, resp.) and \(y_{c}\) denote the last two layers. The encoder for the radar and optical domains are \(f_{r}=y_{c}\circ y_{r}\) and \(f_{o}=y_{c}\circ y_{o}\). We use the outputs of \(f_{r},f_{o}\) as the features passed to the rest of the network. The encoders \(f_{r}(\cdot)\) and \(f_{o}(\cdot)\) use a 3D U-Net backbone architecture [59] consisting of convolution, batchnorm [60], and max-pooling layers with leaky ReLU activations. The 3D operations are applied along both the temporal dimension and the spatial dimensions. This architecture has been used as a state-of-the-art benchmark for a wide variety of prior work in SITS segmentation tasks [9; 44; 58] and offers a relatively simple design that is comparable in performance to other state-of-the-art SITS segmentation architectures that use a separate sequential technique to handle the temporal dimension.