# Self-Supervised Learning across the Spectrum
## 5 Method
### Multimodal Space-Time Contrastive Learning  
Our approach builds upon the key idea of _semantic scene consistency between varying satellite modalities that are captured over the same space and at the same time_. Therefore, our encoder should map image pixels captured over similar space-time to similar representations; while encoding random noncorresponding pixels to differing representations. We incorporate this intuition in our training scheme through contrastive learning. Inspired by recent successes of pixel-wise contrastive learning [56], we propose a pixel-wise contrastive loss that preserves the spatiotemporal structure of our representations for better transfer to downstream pixel-level tasks like semantic segmentation. Figure 4 outlines this approach.  
Prior work [28; 29; 50] on contrastive learning for images often use single image views and perform a variety of data augmentations (e.g., crop, rotate, blur) on a single view. Different augmentations that correspond to the same view are often correlated together as a positive pair for the loss function. However, in the case of satellite images, we benefit from the availability of multimodal data and omit the augmentation step. Each modality captures the same view of Earth at different wavelengths and can be used as a different transformation.  
Similar to prior work [50; 56], we implement a projection head network \(g(\cdot)\) that maps the output of \(f_{r}(\cdot)\) and \(f_{o}(\cdot)\) to the latent space where the contrastive loss is applied. The projection head consists of two successive \(1\times 1\times 1\) 3D convolution layers with batch normalization and LeakyReLU activation. Note that \(g(\cdot)\) is only used during contrastive pretraining and not while training the reconstruction network or the downstream segmentation network. The output of \(g(\cdot)\) is a feature map of the encoded SITS with compressed spatiotemporal dimensions.  
We assign positive pairs as pixels in the feature space with the _same spatial and temporal dimensions_, across different modalities. Pixel pairs with different space or time dimensions are considered negative pairs in our loss, since they correspond to different semantics. We opt to use the InfoNCE loss [61] as our contrastive loss function:  
\[\mathcal{L}_{c}=-\log\left(\frac{e^{\textit{sim}(z_{i},z_{j})/\tau}}{e^{ \textit{sim}(z_{i},z_{j})/\tau}+\sum\limits_{z_{n}\in Z}e^{\textit{sim}(z_{i},z_{n})/\tau}}\right) \tag{2}\]  
Positive pairs \(z_{i}\) and \(z_{j}\) are corresponding space-time pixels in the feature map representations of opposite modalities. \(Z\) is the set of all negative feature map pairs with  
Figure 4: **Multimodal Space–Time Contrastive Learning for SITS.** Our approach operates on the encoded SITS feature maps. Corresponding space–time pixels on the feature map are denoted as positive pairs that the contrastive loss tries to align. Noncorresponding pixel pairs are negative and repelled by the loss.  
anchor pixel \(z_{i}\) in the opposite modality. More broadly, \(Z\) consists of the features that were captured at different spatial locations or different times from \(z_{i}\). The cosine similarity function is defined as \(sim(u,v)=u^{T}\triangledown\|u\|\|v\|\). The temperature hyperparameter \(\tau\) is set to 0.5 by default. The loss is first averaged over all pixels in the first modality's feature map; then, we compute the loss averaged across all pixels in the second modality feature map as anchor pixels. Finally, we average the loss across both modalities together to compute the final contrastive loss per sample in the batch.