SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adam J. Stewart<sup>1</sup>, Nils Lehmann<sup>2</sup>, Isaac A. Corley<sup>3</sup>, Yi Wang<sup>2,4</sup>, Yi-Chia Chang<sup>1</sup>, Nassim Ait Ali Braham<sup>2,4</sup>, Shradha Sehgal<sup>1</sup>, Caleb Robinson<sup>5</sup>, Arindam Banerjee<sup>1</sup>
<sup>1</sup>University of Illinois Urbana-Champaign, <sup>2</sup>Technical University of Munich,
<sup>3</sup>University of Texas at San Antonio, <sup>4</sup>German Aerospace Center,
Microsoft AI for Good Research Lab
The Landsat program is the longest-running Earth observation program in history, with  $50+$  years of data acquisition by 8 satellites. The multispectral imagery captured by sensors onboard these satellites is critical for a wide range of scientific fields. Despite the increasing popularity of deep learning and remote sensing, the majority of researchers still use decision trees and random forests for Landsat image analysis due to the prevalence of small labeled datasets and lack of foundation models. In this paper, we introduce SSL4EO-L, the first ever dataset designed for Self-Supervised Learning for Earth Observation for the Landsat family of satellites (including 3 sensors and 2 product levels) and the largest Landsat dataset in history (5M image patches). Additionally, we modernize and re-release the L7 Irish and L8 Biome cloud detection datasets, and introduce the first ML benchmark datasets for Landsats 4-5 TM and Landsat 7 ETM+ SR. Finally, we pre-train the first foundation models for Landsat imagery using SSL4EO-L and evaluate their performance on multiple semantic segmentation tasks. All datasets and model weights are available via the TorchGeo $^1$  library, making reproducibility and experimentation easy, and enabling scientific advancements in the burgeoning field of remote sensing for a multitude of downstream applications.
On July  $23^{\mathrm{rd}}$ , 1972, the National Aeronautics and Space Administration (NASA) launched Landsat 1. Designed by Virginia T. Norwood, the Multispectral Scanner (MSS) onboard Landsats 1-5 provided invaluable measurements of the Earth's surface in both the visible and infrared spectra. Although she passed away earlier this year, her legacy as "The Mother of Landsat" lives on, as the Landsat program has become the longest-running Earth observation program in history [1].
The Landsat satellite program stretches over 50 years and includes 9 generations of satellites, each with its own set of sensors. Landsats 1-3 carried the Return Beam Vidicon (RBV), an RGB analog camera [2]. However, its lower number of spectral bands and electrical issues meant it was rarely used for research purposes. Instead, the Multispectral Scanner (MSS) onboard Landsats 1-5 was the primary scientific instrument, with a line-scanning and rotating mirror-based camera [3]. Landsats 4-5 also included the Thematic Mapper (TM), with a greater number of spectral bands (from 4 to 7) and finer spatial resolution (from  $80\mathrm{m}$  to  $30\mathrm{m}$ ) [4]. Although it failed to reach orbit and eventually crashed down to Earth, Landsat 6 would have carried the Enhanced Thematic Mapper (ETM), which added a  $15\mathrm{m}$  resolution panchromatic band. Landsat 7 carried the Enhanced Thematic Mapper Plus
<sup>1</sup>https://github.com/microsoft/torchgeo
Figure 1: Spectral wavelengths and spatial resolutions of each band captured by the Landsat sensors used in our study. Numbers are band indices and colors are for visualization purposes only. As each sensor has a different number of spectral bands, spatial resolutions, and wavelengths, it is not possible to train a single one-size-fits-all model.
(ETM+), which upgraded the thermal band from  $120\mathrm{m}$  to  $60\mathrm{m}$  resolution [5]. Onboard Landsats 8-9, the Operational Land Imager (OLI) adds new coastal aerosol and cirrus bands for improved cloud masking, while the Thermal Infrared Sensor (TIRS) adds an additional thermal band [6]. See Figure 1 for a rundown of the spectral wavelengths and spatial resolutions of the primary sensors of interest in our study and Figure 2 for a timeline of when these satellites were active.
In addition to the differences between sensors onboard each satellite, the United States Geological Survey (USGS) distributes several different Landsat products with varying processing levels. Level-1 data, also known as Top of Atmosphere (TOA), are images that have undergone registration against ground control points (GCPs) and orthorectification against digital elevation models (DEMs). These products are particularly useful for cloud masking and other atmospheric applications. Level-2 data includes Surface Reflectance (SR) and other products that have undergone atmospheric correction. These products are useful for a wide range of land surface applications [7].
In recent years, there has been significant activity at the intersection of self-supervised learning (SSL) and remote sensing (RS) due to the wide availability of petabytes of free, unlabeled satellite imagery. An early example of this is Tile2Vec [8], which uses geographic distance between sampled patches and a triplet margin loss for contrastive learning. Geography-Aware SSL [9] instead uses multiple images occurring at the same geospatial location at different points in time to form positive pairs, in combination with an additional subnetwork that tries to guess the latitude/longitude from the learned representation. More recently, masked autoencoders have seen a surge in popularity, including SatMAE [10] and Scale-MAE [11]. Other papers instead focus on dataset curation, allowing generic SSL techniques from computer vision to be applied. Seasonal Contrast (SeCo) [12] creates a new dataset using random Gaussian sampling around cities to diversify the pre-training dataset. SSL4EO-S12 [13] further extends this idea by avoiding overlap between image samples.
Recent review papers [14, 15, 16] targeting the intersection between SSL and RS detail prior work on this topic and offer benchmark results on several models and SSL techniques popular in computer vision, including MoCo [17], SwAV [18], SimSiam [19], Barlow Twins [20], SimCLR [21], and BYOL [22]. Among these, the authors find that although SimCLR and SwAV work well on the ImageNet [23] dataset, MoCo and BYOL tend to learn better representations on RS imagery [15, 16].
Due to their higher spatial resolution and faster repeat period, a lot of recent work, especially in the SSL space, focuses on Sentinel-2 [24, 25, 13], Maxar [9, 26, 27], and Planet [28, 29] satellites. However, many applications are not suited to these satellites. In particular, applications involving
Figure 2: Timeline of all Landsat missions. Crosshatched regions represent partial or complete sensor failure, including electrical issues with RBV on Landsat 1 and SLC-off on Landsat 7. Each bar ranges from launch date to decommissioning date. Note that satellites may be placed in standby mode before the decommissioning date, as is the case with Landsat 4.
long-term trends—including agriculture [30, 31, 32, 33, 34], climate change [35, 36, 37, 38, 39], deforestation [40, 41, 42, 43, 44], and ecology [45, 46, 47, 48, 49]—require a much longer temporal history. While Sentinel-2 has an 8 year history, Landsat's  $50+$  year history makes it essential for monitoring long-term land surface changes. There are an order of magnitude more papers that use Landsat as compared to Sentinel, and Landsat continues to dominate the scientific literature even after the launch of Sentinel-2 and MODIS [50]. The United States Geological Survey (USGS) estimates that Landsat imagery provides users with an annual benefit of $4.2 billion [51].
In this work, we further extend the ideas proposed in SeCo [12] and SSL4EO-S12 [13] to the Landsat imagery domain. Specifically, we improve on the sampling method of the two previously mentioned papers and sample Landsat imagery across the world and across three different sensors and two product levels. We pre-train ResNet [52] and ViT [53] models using SimCLR v1 and MoCo v2 on each combination of sensor and product to produce a suite of pre-trained Landsat foundation models that can be used in downstream tasks. To test these models, we modernize two older datasets based on Landsat 7 and 8 imagery, and create several additional crop classification and land cover mapping tasks. In summary, the contributions of this paper include:
- the first ever SSL dataset for the Landsat family of satellites,
- the largest Landsat dataset in history (1M images per sensor/product, 5M in total),
- modernized and re-released versions of the L7 Irish and L8 Biome cloud detection datasets,
- two new benchmark datasets that can be used across all Landsat sensors and product levels,
- the first ever benchmark datasets for TM and ETM+ SR imagery, and
- the first ever foundation models pre-trained on Landsat imagery.
Importantly, all of these SSL techniques, datasets, and pre-trained models are distributed via the TorchGeo library [54], allowing for ease of use, experimentation, and reproducibility.
In this section, we detail our methodology behind the collection of the SSL dataset we create, including differences from prior work. We also introduce the existing and newly created benchmark datasets we use to evaluate the representations learned by our pre-trained models.
2.1 SSL4EO-L pre-training dataset
For our SSL pre-training dataset, we extend the methodology introduced by Manas et al. [12] and refined by Wang et al. [13]. Specifically, we iterate over the following steps:
1) sample one of the 10K most populous cities in the world [55] uniformly at random;
2) sample a  $264 \times 264$  px ( $7.92 \times 7.92 \mathrm{~km}$ ) patch from a Gaussian distribution with a  $50 \mathrm{~km}$  standard deviation centered around the centroid of the sampled city;
3) ensure the patch does not overlap with any existing sampled patches;
4) ensure that there exist 4 patches of imagery from 4 different seasons—each selected from a 60-day window centered about the vernal and autumnal equinoxes and the summer and winter solstices (within a 2-year window)—with less than  $20\%$  cloud coverage;
5) ensure that none of these patches contain nodata pixels;
6) if the previous three criteria are met, download the imagery corresponding to the patch.
If any step in this algorithm fails (there is overlap, or a location does not have a set of 4 cloud-free, nodata-free images), the sample is skipped and we start over at step 1. This algorithm is designed to maximize the diversity of images in the dataset, relying on the assumption that most of the diversity in land cover is centered around large cities, with a gradual transition between urban, suburban, farmland, and forest. Uniform sampling would instead result in images that are  $70\%$  ocean,  $10\%$  desert, and  $9\%$  forest, resulting in very little dataset diversity [12]. Note that this sampling strategy does result in decreased sampling from regions with persistent cloud cover (tropical rainforests) or lower populations (desert, taiga, tundra, and polar biomes). By sampling different points in time, we allow seasonal differences to act as natural forms of data augmentation during contrastive learning.
Differences between our sampling strategy and the one used by SSL4EO-S12 are as follows. SSL4EO-S12 used Euclidean distance between patch centroids and a grid heuristic to detect overlap between patches. This method has an  $\mathcal{O}\left(N^2 /M\right)$  average run-time complexity, where  $N$  is the total number of samples and  $M$  is the number of grid cells. We replace this with an  $\mathcal{O}\left(N\log N\right)$  R-tree [56], removing the  $1 - 3\%$  overlap reported by Wang et al. [14] due to use of this grid heuristic. Among the cloud-free images in the aforementioned time windows, we sort by cloud cover instead of date to provide the best possible image patches. We also skip patches containing nodata pixels due to sampling near the border of a scene, which we found to be prevalent (on the order of  $25\%$ ) in prior datasets. We found it necessary to increase the cloud coverage threshold from  $10\%$  to  $20\%$  due to the larger patch size (Sentinel-2 has a  $10\mathrm{m}$  resolution, but Landsat has a  $30\mathrm{m}$  resolution, resulting in patches that cover  $9\times$  the area) and avoidance of nodata pixels. Finally, since the resolution of most bands are the same, we resample all thermal and panchromatic bands to a  $30\mathrm{m}$  resolution, allowing all bands to be concatenated into a single file.
We download all data from Google Earth Engine (GEE) [57], with a total of 250K locations, each sampled at 4 different seasons, for a total of 1M unlabeled image patches per sensor/product and 5M in total. Each image is  $264 \times 264$  px, corresponding to  $7.92 \times 7.92$  km at  $30\mathrm{m / px}$  resolution. There are separate datasets for TM TOA, ETM+ TOA, ETM+ SR, OLI/TIRS TOA, and OLI SR. We decided not to include RBV and MSS sensors due to the limited data availability on GEE and the fact that it is not possible to create a benchmark dataset for these sensors due to their age. Since TM and ETM+ use the same sensor for SR bands, we did not create a separate dataset for TM SR. For similar reasons, there is a single dataset for OLI/TIRS and OLI-2/TIRS-2. TM data is collected from 4 different seasons in 2009-2010, as the TM sensor failed in November 2011. ETM+ data is collected from 2001-2002, as the scan line corrector (SLC) failed in May, 2003, resulting in images with significant nodata pixels. OLI/TIRS data is collected from 2021-2022. See Figure 3 for a map of the geographical distribution for each sensor. Note that it is not possible to sample high latitudes due to lack of winter imagery.
All TOA and SR datasets represent a parallel corpus (the TOA and SR images are taken at the same locations and dates). Due to differences in collection years and cloud coverage/nodata pixels, it was not possible to create a parallel corpus between sensors. However, approximately  $50\%$  of TM and  $\mathrm{ETM+}$ ,  $40\%$  of TM and OLI/TIRS, and  $40\%$  of  $\mathrm{ETM+}$  and OLI/TIRS images are sampled from the same location, allowing for multimodal data fusion studies. The official scale factors suggested
(a) OLI/TIRS
(b) TM
(c) ETM+
Figure 3: Geographical distribution of the SSL4EO-L dataset, including the (a) Landsat 8-9 OLI/TIRS, (b) Landsat 4-5 TM, and (c) Landsat 7 ETM+ splits. Surface reflectance (SR) and top of atmosphere (TOA) products are sampled from the same locations per sensor.
by the USGS to map between Level-1 and Level-2 Landsat imagery $^{2}$  and the visualization range recommended by GEE for each sensor are used to map from float32 to uint8. The resulting datasets are 274–385 GB when compressed and can be downloaded from Hugging Face $^{3}$  using TorchGeo.
2.2 Dataset archaeology
In order to benchmark the ability of our learned representations to transfer to downstream applications, we require curated benchmark datasets for evaluation. Although there exist  $\sim 10$  semantic segmentation datasets for OLI/TIRS TOA, an extensive literature review found almost no benchmark datasets for other sensors, products, or tasks. This is due to both their age (deep learning was not commonplace in the field of remote sensing until recently) and the fact that semantic segmentation is the primary task for which lower resolution satellite imagery is used.
A single classification dataset, Statlog [58], was found for the MSS sensor. However, this dataset is composed of  $3 \times 3$  px images, making it unsuitable for evaluation of CNN and ViT backbones. For the task of semantic segmentation for cloud cover, three ETM+ TOA datasets were found: L7 SPARCS [59], L7 Irish [60, 61], and L7 Scaramuzza [62]. Each of these datasets also has a corresponding dataset for OLI/TIRS TOA (L8 SPARCS [63, 64], L8 Biome [65, 66], and L8 Scaramuzza [67]), making it possible to compare learned representations across sensors. No benchmark datasets for TM or ETM+ SR were ever found. The L7 SPARCS dataset, while thought to be lost to time, was eventually recovered from a hard drive found in the closet of one of the dataset's authors. The majority of the aforementioned cloud segmentation datasets are official datasets used by the USGS to validate their cloud detection algorithms. Among these datasets, we chose to use L7 Irish and L8 Biome due to their larger size and greater number of citations.
2.2.1 L7 Irish dataset
The L7 Irish dataset, originally selected by Irish et al. [68] and later digitized by Scaramuzza et al. [61], is a validation dataset for cloud cover assessment algorithms composed of 206 Landsat 7 ETM+ Level-1G scenes and manually generated cloud masks divided between 9 unique biomes. Each scene is a 9-band, roughly  $8000 \times 8000$  px multispectral image with  $30\mathrm{m / px}$  resolution. Cloud masks consist of 5 classes: 1) fill, 2) cloud shadow, 3) clear, 4) thin cloud, and 5) cloud.
There are 2015 [69] and 2019 [60] versions of this dataset available for download. Unfortunately, both versions have numerous issues that make them difficult to use for evaluation. The 2015 version contains 1 scene with a corrupted thermal band file, 2 scenes that are missing masks, 1 scene with an
$^{2}$ https://www.usgs.gov/faqs/how-do-i-use-scale-factor-landsat-level-2-science-products
<sup>3</sup>https://huggingface.co/torchgeo
inconsistent filename format, and the documented class labels do not match the actual class labels used. Additionally, there is no way to programmatically download the entire dataset. All 206 files must be manually downloaded, one at a time, with a limit of 6 parallel downloads, requiring 3-4 hrs of constant supervision and clicking each link every  $5\mathrm{min}$ . The 2019 version has even more issues, including 5 scenes with corrupted thermal band files, 1 scene missing geolocation, 6 scenes with inconsistent filename formats, and inconsistent thermal band resolutions. Although  $17\%$  of masks matched the documented labels, the other  $83\%$  of masks use a completely different mapping, with both clear and fill classes mapped to the same value.
In order to use this dataset for evaluation, we start with the 2015 version and use scenes from the 2019 version to replace corrupted images and missing masks. We correct the class mapping of copied masks and copy the fill pixels from the images to the masks. We convert all images to Cloud Optimized GeoTIFFs (COGs), resample to  $30\mathrm{m}$  resolution, and stack them into single multi-band files with consistent filenames. The compression algorithm used by COGs resulted in a dataset that is  $33\%$  of the original size and therefore faster to download and load from disk. The final ML-ready dataset is available on Hugging Face and can be automatically downloaded using TorchGeo.
2.2.2 L8 Biome dataset
The L8 Biome dataset, created by Foga et al. [65], is a validation dataset for cloud cover assessment algorithms consisting of 96 Landsat 8 OLI/TIRS Level-1T scenes and manually generated cloud masks evenly divided between 8 unique biomes. Each scene is an 11-band, roughly  $9000 \times 9000$  px multispectral image with  $30\mathrm{m / px}$  resolution. Cloud masks consist of the same 5 classes as L7 Irish.
Comparatively, L8 Biome has fewer issues than L7 Irish. The masks lack geolocation, but we can copy this from the image files. While the dataset can be programmatically downloaded, it requires scraping a webpage for 96 different URLs for each scene. We convert the raw uint16 images to uint8 to match L7 Irish, and create compressed COGs of all files, resulting in a dataset  $9\%$  of the original size. We resample all images to  $30~\mathrm{m / px}$  resolution and stack them in single multi-band files. The dataset is available on Hugging Face and can be automatically downloaded using TorchGeo.
2.3 SSL4EO-L benchmark dataset
As there are no existing benchmark datasets for TM or ETM+ SR, we need to design our own. Crucially, we want a single benchmark dataset that can be used for a consistent comparison across all 5 sensors/products for which we are pre-training models. We create our own land cover classification datasets based on NLCD [70] and CDL [71] masks, described in more detail below. They are the only large, Landsat-based semantic segmentation masks with a long enough history to benchmark foundation models for historical satellites.
Our sampling strategy is similar to the one used for our pre-training dataset, with a few differences. As CDL only exists for the continental U.S. (CONUS), we restrict our sampling strategy to CONUS. To achieve maximum coverage, especially in lower population regions where agriculture is most prevalent, we replace the city-centered Gaussian distribution with a uniform sampling distribution. We choose a single 60-day window centered around August  $1^{\mathrm{st}}$  when crop types are easiest to distinguish. As CDL data is not available before the  $\mathrm{ETM + }$  SLC failure, we do not exclude nodata pixels for this sensor. Additionally, nodata masks are copied from SLC-off imagery to masks so as to avoid penalizing models for making incorrect predictions where there is no data. The 2019 NLCD and CDL datasets are used for  $\mathrm{ETM + }$  and OLI/TIRS evaluation since 2019 is the most recent year for which both datasets exist. The 2011 datasets are used for TM since 2011 is the most recent year for which both Landsat 5 and NLCD/CDL overlap. These years are different than the years collected for our pre-training dataset, allowing us to accurately measure performance on images that the pre-trained model has never seen before.
The resulting dataset consists of 25K Landsat, NLCD, and CDL triplets, converted from float32 to uint8 using the same scaling as above. All images have the same resolution and dimensions as the pre-training dataset. The datasets form a parallel corpus between TOA and SR products, and have approximately  $85\%$  spatial overlap across sensors, although not necessarily during the same year, allowing for multimodal data fusion studies. All datasets are available for download from Hugging Face using the TorchGeo library, making it easy for other researchers to compare against our preliminary benchmark results.
NLCD The National Land Cover Database (NLCD) [70] is a land cover product produced every 2-3 years by the USGS, in collaboration with the Multi-Resolution Land Characteristics (MRLC) consortium. The dataset spans the entire U.S. from 2001-2019. The final products are generated at a  $30\mathrm{m}$  resolution by random forest models trained on spectral, spatial, temporal, and ancillary data [72, 73, 74]. We use the 21 class version, with an estimated overall accuracy of  $77.5 \pm 1.0\%$  [75].
CDL The Cropland Data Layer (CDL) [32] is an annual land cover product produced by the U.S. Department of Agriculture (USDA) National Agricultural Statistics Service (NASS) focusing on crop classification. Although the dataset is available starting in 1997, full CONUS coverage is not available until 2008. The dataset consists of 134 classes, primarily for agricultural crops grown in the U.S. Labels are generated at a  $30\mathrm{m}$  resolution using a decision tree classifier. The most common crop classes are estimated to have an accuracy of  $85 - 95\%$  [32]. All non-agricultural classes are taken from NLCD, and should be considered to have a similar accuracy.