# Self-Supervised Learning across the Spectrum
## 4 Problem Setup: Satellite Imaging with Multiple Modalities  
Our work is situated in the emerging context where different satellite constellations capture Earth imagery in different frequency bands. We seek to extract spatiotemporal insights from these data. A majority of the satellites capture optical images that passively monitor the reflections of sunlight off the Earth's surface. These optical images are often multispectral, including imaging bands outside the standard visible red, green, and blue channels. However, a key disadvantage of such imagery is that optical satellite images are often occluded by clouds (Figure 3) and are easily obscured in low-lighting conditions, such as night and fog [18; 19; 39; 40]. Some satellites are equipped with radar imaging that works by actively transmitting pulses of radio waves and measuring the reflectance of these radio pulses to produce radar images. These radio waves utilize a longer wavelength than optical images and are typically better at monitoring certain aspects of the surface, such as moisture and topology. However, the resolution of radar imagery is lower than that of optical images. Satellites are typically equipped with either optical or radar imaging modalities, _but not both_[20]. Therefore, images in optical and radar SITS cannot be perfectly aligned in time. Each image captured by satellites is georeferenced, i.e., we can extract per-pixel geographic coordinates. This allows us to spatially align images even when captured on different satellites. However, leveraging the temporal aspect of SITS data poses some challenges. First, images in SITS, unlike videos, are not taken at regular intervals. Images are taken over a location only when a satellite orbits over that location, meaning the time between images in SITS is irregular based on the satellite's orbit. Second, for multimodal SITS, different sensing modalities are often located on different satellites, meaning that images of SITS of different modalities are not only unaligned in time, but they can also result in time series of vastly different lengths. S4's primary task is to use the ample amount of unlabeled imagery collected by satellites for cross-modal self-supervision. Our formulation builds upon the key idea of _pixel-level semantic consistency between multimodal images captured over the same location at roughly the same time._ We propose a new training objective that encourages the similarity of corresponding space-time features across modalities while maximizing the distance between the features corresponding to either different locations or different times. Though different modalities have certain distinctions like differing spectral ranges, the _semantic representation of the underlying scene should be agnostic to both wavelength and noise_, e.g., cloud cover (for optical) or capture angle (for radar), and thus S4 can be used to achieve a course-grained alignment of the multimodal, multitemporal features beneficial for self-supervised learning. By leveraging these natural structural characteristics of SITS, our approach extracts a more informative representation that limits the impact of modality-specific noise. **Notation and Setup** We consider the respective radar and optical image modalities, \(\mathcal{X}_{r}\subset\mathbb{R}^{(T_{1}\times C_{1}\times H\times W)}\) and \(\mathcal{X}_{o}\subset\mathbb{R}^{(T_{2}\times C_{2}\times H\times W)}\), where \(T_{i}\), \(C_{i}\), \(H\), and \(W\) are the number of images in the time series, number of image channels, image height, and image with dimensions, respectively. During training, we assume access to \(N\) SITS pairs \(\{(x_{o}^{l_{n}},x_{r}^{l_{n}})\}_{n=1}^{N}\in(\mathcal{X}_{o}\times\mathcal{X}_{r })^{N}\), where \(l_{i}\) corresponds to the location where the SITS was captured.