Small data problems in deep learning applications with remote sensing: A review

Researchers and engineers have increasingly used Deep Learning (DL) for a variety of Remote Sensing (RS) tasks. However, data from local observations or via ground truth is often quite limited for training DL models, especially when these models represent key socio-environmental problems, such as the monitoring of extreme, destructive climate events, biodiversity, and sudden changes in ecosystem states. Such cases, also known as small data problems, pose significant methodological challenges. This review summarises these challenges in the RS domain and the possibility of using emerging DL techniques to overcome them. We show that the small data problem is a common challenge across disciplines and scales that results in poor model generalisability and transferability, yet this has not been investigated in a structured way. We first introduce ten emerging DL techniques: transfer learning, self-supervised learning, semi-supervised learning, few-shot learning, zero-shot learning, active learning, weakly supervised learning, multitask learning, process-aware learning, and ensemble learning; we also include a validation technique known as spatial k-fold cross validation. These techniques have shown promising potential in other scientific disciplines, but have been rarely applied in the RS domain. We also provide guidance on which learning technique to use in various cases, which helps to create a more methodologically robust DL application (and a greater number of them) that can be used to tackle socially important problems with limited data. Introduction Over the last decade, Artificial Intelligence (AI) technologies, especially Machine Learning (ML) and Deep Learning (DL), have been increasingly used for understanding and predicting human-environment interactions. ML is a subset of AI that implements algorithms which use data to learn how to perform a specific task without being explicitly programmed. DL is a subset of ML that focuses on training deep neural networks capable of implicit feature extraction from unstructured data, such as images, text, and sound. Scientists have actively employed DL for image processing and data analysis, recently providing innovative solutions in the field of Remote Sensing (RS) to detect and classify objects on Earth. This study defines RS as the use of satellite and aircraft-based sensors. The expanding field of RS provides an abundance of data streams from numerous sources. This, combined with the growing array of available data products, delivers a wide range of data that is useful for addressing various problems. Among them, Landsat, has been operational since the early 1970s and provides a unique long-term record of satellite imagery with a 30-metre spatial resolution. The Copernicus programme’s Sentinel-2 system generates data with a 10-metre spatial resolution, offering a balance between spatial detail and data continuity as well as radar imagery based on the Sentinel-1 mission. One recently launched hyperspectral mission, the Environmental Mapping and Analysis Program (EnMap), stands out with over 200 spectral bands and a 30-metre spatial resolution; this offers unique opportunities for researchers to map ecosystems and their changes in detail. In addition, commercial platforms, such as SkySat, provide extremely high-resolution data with a spatial resolution of less than one metre. Together, these diverse RS platforms contribute to a more comprehensive understanding of the Earth’s surface across different scales and domains. For this reason, these RS products are widely used to study local and regional environmental problems, including agricultural productivity, the water quality of lakes and ponds, the ecological patterns of forests and grasslands, and damage to natural, cultivated, and inhabited land through extreme weather events. Since the spatial and temporal resolution of RS products is likely to continue increasing, DL applications are expected to become even more popular for solving fine-scale local issues where each local site has its own unique conditions and context. Since DL algorithms have fewer inductive biases but larger parameter counts than conventional ML algorithms, DL models normally require a large amount of data for training. DL methods usually learn from raw data and skip manual feature engineering steps; this means that human efforts are not needed to quantitatively measure some attributes from the data. For example, DL algorithms can learn from image data directly, instead of using the extracted shape and size of an object in an image. When sufficient data is available, DL methods can automatically extract the meaningful features from low to high levels for prediction. However, although raw data of common events is generally abundant, the lack of sufficient labelling information makes the collection and preparation of a large reference dataset a persistent challenge for many RS applications. Moreover, certain scenarios also lack available reference data. For instance, biodiversity monitoring needs a large number of human observers well trained in taxonomic classification, which often prevents observation campaigns from generating datasets large enough for sound DL applications. Furthermore, anomaly events such as climate extremes and disease outbreaks are too rare for researchers to acquire sufficient data coverage. Their sample size is often as small as n = 1–300, which is usually insufficient for DL application . The gap between the large data availability of RS imagery and the small data availability of several important real-world environmental problems (referred to as the “small data problem”) is a very common challenge. It is hard to acquire the ground-truth response labels associated with the input features. This makes sense, because the goal of most of these studies is to develop a model designed to predict a specific response variable from the various observed input features. However, traditional DL training methodologies require a large initial set of labelled data to train predictive models. It is increasingly clear that this is an emerging problem for AI, and researchers have proposed several novel DL techniques that require less labelled data (e.g., transfer learning and self-supervised learning). However, to the best of our knowledge, the small data problem has not been systematically tackled as an emerging scientific challenge in the RS domain. In this review, we summarise the current research on the small data problem in RS (particularly as it relates to DL) and suggest promising DL techniques to address this problem. First, we explore how the small data problem can be defined. Second, we describe a few common elements of the previous studies. Third, we present the advantages and disadvantages of using a small dataset. Finally, we offer a set of practical recommendations about how RS scientists can better implement DL techniques to fully make use of a small dataset. We believe that the small data problem is a common – but still unsolved – issue for recent RS applications, and therefore, this review should serve as a valuable resource for supporting RS and DL applications in scientists’ and policymakers’ attempts to address a wide range of environmental problems. What is the small data problem? We argue that a dataset can be considered large (not small) when the dataset consists of >100,000 annotated samples, or when it covers the entire probability distribution in a high-dimensional space. In this case, model generalisability and transferability are expected to be high. For example, there are several free large datasets that can be used for DL: the ImageNet dataset, containing over 14 million annotated images, the Common Objects in Context (COCO) dataset, containing 330K images, 1.5 million object instances, and 80 object categories, and the OpenImages dataset, containing over 9 million images. These datasets can be used for training a large DL model with thousands to millions of parameters. In the RS domain, land use / land cover classification would be a typical example. In contrast, data is more likely to be regarded as small (or not large enough) when the dataset consists of <1,000 annotated samples, the dataset covers the distribution poorly, or the number of samples is expected to be insufficient when using DL to find meaningful features. This is a frequently occurring situation, but it can be a significant challenge for training deep neural networks. A relatively small dataset can negatively affect the performance of a DL model due to overfitting, which is when a model performs well with the training data but poorly on new, independent testing data. This therefore results in low levels of model generalisability and transferability. A common case within the RS domain (but particularly relevant) is that the data can be “extra-small”, meaning that the dataset consists of just 1–10 annotated samples (e.g., historical natural disasters and disease outbreaks). The size would be sufficient for human beings to start guessing what features can uniquely describe the target, but it would not be sufficient for automated, implicit DL feature extraction. In the DL domain, the Tiny ImageNet Dataset (also known as MicroImageNet) contains 500 images for each class (of 200 classes), indicating that DL scientists regard this level of data size as small. According to the articles we reviewed in the following section, the majority of the studies targeted classification of a few types, and many of them collected less than 500 images for each class (e.g., refs.). There should not be a strict divide between “small” and “large” when training DL models, because the size of the dataset required may depend on various factors such as the complexity of the task and the number of features in the data. Typically, the challenges stemming from a limited amount of labelled data increase with system complexity, the rarity of observations (e.g., endangered biol