Reviewerly/exHarmony huggingface dataset:
Label Noise Robust Image Representation Learning Based on Supervised Variational Autoencoders in Remote Sensing

Due to the publicly available thematic maps and crowd-sourced data, remote sensing (RS) image annotations can be gathered at zero cost for training deep neural networks (DNNs). However, such annotation sources may increase the risk of including noisy labels in training data, leading to inaccurate RS image representation learning (IRL). To address this issue, in this paper we propose a label noise robust IRL method that aims to prevent the interference of noisy labels on IRL, independently from the learning task being considered in RS. To this end, the proposed method combines a supervised variational autoencoder (SVAE) with any kind of DNN. This is achieved by defining variational generative process based on image features. This allows us to define the importance of each training sample for IRL based on the loss values acquired from the SVAE and the task head of the considered DNN. Then, the proposed method imposes lower importance to images with noisy labels, while giving higher importance to those with correct labels during IRL. Experimental results show the effectiveness of the proposed method when compared to well-known label noise robust IRL methods applied to RS images. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/RS-IRL-SVAE.
Synthetic Aperture Radar Image Enhancement Based On Residual Network

Spatial resolution of synthetic aperture radar (SAR) is a vital index to evaluate the performance of its observed image. However, high spatial resolution of SAR is achieved at the cost of system resources. Therefore, super-resolution methods can be applied in SAR systems to improve their spatial resolution without system resource increases. In this paper, we propose a new residual network-based structure for super-resolution of SAR images. The proposed method adopts the structure of global residuals and adds several convolutional layers before and after the residual module to take into account the depth and width of the network. The simulation results show that the proposed method is effective as the visual effect and data evaluation.
XAI for Early Crop Classification

We propose an approach for early crop classification through identifying important timesteps with eXplainable AI (XAI) methods. Our approach consists of training a baseline crop classification model to carry out layer-wise relevance propagation (LRP) so that the salient time step can be identified. We chose a selected number of such important time indices to create the bounding region of the shortest possible classification timeframe. We identified the period 21st April 2019 to 9th August 2019 as having the best trade-off in terms of accuracy and earliness. This timeframe only suffers a 0.75 % loss in accuracy as compared to using the full timeseries. We observed that the LRP-derived important timesteps also highlight small details in input values that differentiates between different classes and possibly offers links to physical crop growth milestones.
Deep Continuous Matching Network for more Robust Multi-Modal Remote Sensing Image Patch Matching

Due to the powerful feature extraction capabilities of deep neural networks, traditional approaches are gradually replaced by deep learning approaches for image matching tasks. For multi-modal image patch matching, the deep model should mainly learn the modality-invariant features. For multi-modal images with rotation transformation (RT), the deep model should learn the modality-invariant features and rotation-invariant features simultaneously. However, the performance of the latter trained model is degraded for the former task. The main reason is that the modality invariance of the features degenerates. This paper proposes a deep multi-modal remote sensing image matching network (DCMNet) that combines descriptor learning and continuous learning to solve this problem. Firstly, DCMNet is trained for learning modality-invariant features in multi-modal image patch matching. Then, DCMNet is optimized for multi-modal image patch matching with RT. In the later learning process, we reduce the change of important parameters for the modality-invariant features learning. Experiments demonstrate the effectiveness and robustness of DCMNet in alleviating the modal invariance degradation problem of features.
A Dual-Branch Awareness Network for Small Object Segmentation in Large-Scale Remote Sensing Scenes

The more detailed and accurate earth observation has been made driven by the advancement of satellites and sensors optical photography technology, which poses both a challenge and an opportunity to small object segmentation task. However, the inherent difficulty and inadequate consideration still make small object segmentation task inevitably encounter a performance gain bottleneck. In this paper, we consider the longstanding but underestimated challenges in this task and give a point-to-point solution to response them. Specifically, we introduce a discriminative structure, i.e., a dual-branch awareness network for small object segmentation, named DASNet. In this structure, we propose the small object activation branch and the fuzzy refinement branch to avoid the negative influence of redundant background and ensure the small object segmentation accuracy, respectively. These two branches work collaboratively to mimic the process of human visual perception on small object. Finally, we propose a hierarchical unbiased loss to eliminate the bias against small objects in the regression process. Extensive experiments demonstrated that DASNet is competitive against some advanced methods for small object segmentation.
Filtered Residual Compression for Satellite Images

Learned image compression neural networks have difficulties adapting to certain satellite image characteristics, especially high frequencies that disappear at a high bit-rate in the blur generated in the reconstruction. To answer this problem we describe a joint end-to-end trainable neural network. It is separated into a general compression network and a smaller specialised network. We train a specialized network to compress the residual part of the image to best preserve the high-frequency details present in the satellite images. The proposed model achieves higher rate-distortion performance than current lossy image compression standards and also manages to retrieve details previously poorly reconstructed.
DFRI:Detection and Fine-Grained Recognition Integrated Network for Inshore Ship

Remote sensing ship detection is of great importance for the applications of maritime transport and precision guidance. However, the background interference on land, the large similarity between ship classes, and the arbitrary orientation of ships all pose serious challenges for inshore ship detection. To solve the problem of complex and diverse land backgrounds and sea conditions, we use the contextual attention module to enhance the ship target features. In addition, a rotation-invariant feature extraction and fusion module is introduced to make the features directionally invariant and increase the inter-class distance by fusing global features with local features. Finally, the feature decoupled module is redesigned to resolve the conflict between target detection and fine-grained recognition. Experimental results show that our proposed integrated model for inshore ship detection and fine-grained recognition improves mAP by about 6% compared to the current baseline model.
Continual Self-Supervised Learning in Earth Observation with Embedding Regularization

Continual Self-Supervised Learning (CSSL) is a promising approach for intelligent systems that address the challenge of learning in scenarios with limited data, mirroring real-world conditions. However, CSSL remains relatively unexplored, especially in the context of Earth Observation (EO). In this paper, we investigate the problem of CSSL in remote sensing (RS), focusing on leveraging satellite and aerial imagery to develop systems that can continuously adapt and learn with minimal human intervention in data preparation. Specifically, we tackle the task of semantic segmentation, which has diverse applications in RS. Building upon existing work in the domain, we propose a novel algorithm called Continual Barlow Twins with Embedding Regularizer (CBT-ER). To evaluate the effectiveness of our approach, we conduct experiments on three heterogeneous datasets (i.e. Potsdam, DFC2022, SEN12MS). To ensure robust experimentation, we vary the availability of data labels (10%, 100%) and compare our approach against different baselines, showing encouraging performance.
A SAR-SAR Template Matching Based on Autoencoders and Multi-Level and Multi-Rotation Feature-Space Correlation

This work presents an unsupervised Deep Learning (DL)-based method for the matching of SAR images in case of high variability of the data. The method is based on an autoencoder trained in an unsupervised manner, with the SAR image in input and output given at the same time. From this architecture, the encoder is detached and used as feature extractor to obtain a set of feature maps at different depths to be used for the comparison. The correlation is applied to these feature maps to get the point of maximum correspondence among the image and the template to be found. The DL model has been trained with a heterogeneous dataset, with images coming from different SAR sources, in order to make it generalised and to make it work in a wide range of scenarios. The method has been tested both with similar and completely different images, showing good performances even in the most critical case.
Detection of Heavy-Polluting Enterprises from Optical Satellite Remote Sensing Images

Heavy-polluting enterprises burn fossil fuels to release large amounts of greenhouse gases, causing severe pollution worldwide. Heavy-polluting enterprises have a significant responsibility for carbon emissions, and more than 130 countries have set or are considering targets for achieving net-zero carbon emissions by 2050. Assessing these enterprises can provide data support for carbon emissions and aid in evaluating industry’s economic development. In view of the problem that the existing research data is not comprehensive and the generalisation ability is week. To address this issue, we construct a high-resolution remote sensing image dataset of global heavy-polluting enterprises and use the classic target detection network SSD, Faster R-CNN and YOLOv3 for training, testing and evaluation. The experimental results findings indicate that the SSD network is particularly well-suited for object detection of heavy-polluting enterprises in the remote sensing domain.
Visual Exploration of Satellite Image Time Series

Satellite image time series are a worthwhile source of information for a broad range of applications, especially in the context of future global challenges. The challenge to discover correlations, patterns or anomalies would be eased if the data analysts might benefit of visualization tools enabling them to grasp, briefly, the characteristics of the time series. Responding to these needs, this paper proposes a graphical user interface focused on SITS visualization. While overcoming the current software limitations we developed a Python GUI interface enabling remote sensing scientists who use Python for their research, to visually analyze the satellite image time series in the same environment.
Spatio-Temporal Data Harmalization Using Knowledge Graph for Soil Health Management

Spatio-temporal data harmonization is crucial for effectively managing soil health in sustainable agriculture. In the case of Indian soil, known for its diversity and complexity, integrating and analyzing different soil data sources presents significant challenges. This research proposes a novel approach that utilizes knowledge graphs to address these challenges in soil health management.This paper presents a knowledge graph that captures domain-specific information about Indian soil, such as soil properties, climatic conditions, land use patterns, and agricultural practices. This knowledge graph serves as a unified framework for integrating diverse data sources, such as soil survey lab reports, sensor data, and agricultural databases. An architecture for spatiotemporal data harmonization is presented in this paper, which includes data acquisition, pre-processing, knowledge graph construction, embedding techniques, alignment, harmonization, and integration. In the context of soil health management, it facilitates the integration and analysis of diverse soil data sources.
An Imbalanced Signal Modulation Classification And Evaluation Method Based On Synthetic Minority Over-Sampling Technique

Imbalanced signal modulation classification is a challenging problem in wireless communication. In this paper, we propose an overall framework from data generation to generated data qualitative analysis. It generates data based on Synthetic Minority Over-Sampling Technique (SMOTE) and evaluates the generated data in terms of amplitude, distribution, and classification validity, namely SMOTE-ADC. The framework aims to generate synthetic data for the minority class to rebalance the dataset and improve the classification performance. SMOTE-ADC creates and defines a matrix of distances between eigenvalues by considering the eigenvalues and their relationships. By comparing the variability of neighboring feature vectors and defining the weights of the distance matrix to make the generated data more close to the real data. For the evaluation of generated data in SMOTE-ADC, we propose three methods to analyze the effect of generated data, comparing the amplitude, distribution, and classification effectiveness of generated data and original data, respectively. Experimental results demonstrate that our approach effectively mitigates the impact of data imbalance and enhances the accuracy of signal modulation classification.
Multimodal self-supervised learning for semantic analysis of PolSAR imagery

The automatic prediction of semantic maps, e.g. for land use/cover, is often addressed via supervised learning approaches that rely on large training datasets. Obtaining the required reference annotations for a large amount of data is tedious and time consuming work, in particular for sensors such as synthetic aperture radar. Self-supervised learning aims to mitigate this issue by leveraging the large amount of usually available unlabelled data by constructing a pretext task that does not require annotations. We evaluate DINO, a recent self-supervised learning approach, and show that it outperforms segmentation models that have been initialized randomly or by weights obtained from pretraining on ImageNet. Furthermore, we show that the inclusion of multimodal data during the pretraining phase improves results significantly.
Rotated Ship Detection Based On Dense Points in High Resolution Remote Sensing Images

The rapid development of remote sensing technology has provided convenient conditions for obtaining abundant research data. The use of visible light remote sensing images for ship detection has profound significance in the fields of port management, maritime rescue, and military investigation. Our paper focuses on the shortcomings of current way of ship positioning expression and uses deep learning to study the rotated ship detection task. A kind of ship detection algorithm based on dense points related to the position of ships is proposed to solve the problem of angle boundary and vertex sorting ambiguity in current rotated object detection methods. Our method achieves 92.4 mAP on the HRSC2016 dataset, ranking among the top in similar studies.
Evaluation of Resampling Techniques to Provide Better Synthesized Input Data to Super-Resolution Deep Learning Model Training

Hyperspectral images often have low spatial resolution due to the sensor sizes required to capture the required spectral response. Super-resolution (SR) techniques try to mitigate this by injecting more detail in the upscaled image, either with numerical methods or deep learning and Convolution Neural Networks. In the deep learning methods, the models learn image details by inferring a high-resolution (HR) image from a synthetic low-resolution (LR) image that simulates the natural degradation of sensors by applying resampling (to reduce the image detail) and noising (to add small errors and interference). Often disregarded in the literature, the resampling method applied to generate the synthetic image can impact greatly the deep learning model training. This work, evaluate several resampling techniques to measure this impact using the Harvard hyperspectral dataset. Results showed that the Lanczos filter was the best among eight other resampling methods. The Nemenyi and Friedman ranking statistical tests also indicated that the Cubic-Spline, Bicubic, and RMS achieved good results.
Improved Self-Supervised Texture Recognition of Mastcam Images by Eliminating Mixed Terrain and Range Patches

Homogeneity within a terrain image is crucial for the scientific categorization of the image. Images consisting of more than one terrain class are irrelevant to geologic tasks such as classification, and novelty detection which require granular terrain categories. Further, images containing far-away geological objects are less relevant for scientists interested in studying rock types for Martian paleoclimate and habitability. In this work, we use image segmentation to identify and eliminate images that show multiple terrains, and depth estimation to exclude images taken from a distance beyond a certain range. We then show an improvement in the performance of deep clustering of Martian terrain images qualitatively and discuss the resulting retrieval performance that helps scientists rapidly categorize geologic terrain images.
Non-Local SAR Image Despeckling Based on Sparse Representation

Speckle noise is an inherent problem of synthetic aperture radar (SAR) images, which not only affects the acquisition of SAR image information, but also increases the difficulty of image interpretation and analysis, and greatly reduces the efficiency of image segmentation and feature classification. Based on the nonlocal idea of SAR-BM3D and the concept of sparse representation, a new synthetic aperture radar (SAR) image speckle noise suppression algorithm is proposed. The algorithm follows the framework structure of SAR-BM3D, performs logarithmic conversion after the coarse filtering in the first step, replaces the three-dimensional similar block group after the similar block matching in the second filtering step with a two-dimensional matrix stacked by columns, and replaces the local linear minimum mean square error (LLMMSE) denoising with K-SVD sparse representation image denoising for speckle noise suppression of SAR images. The experimental results show that the non-local denoising method based on sparse representation achieves better results in terms of evaluation metrics and visual perception quality compared with several existing advanced reference techniques.
SSL for Large-Scale Mining Site Segmentation

Mining plays a crucial role in driving industrial development, but unfortunately, it also gives rise to significant environmental pollution. With a keen awareness of this issue, we introduce a small dataset comprising Sentinel-2 RGB images depicting 150 large-scale Chilean mining sites, along with corresponding grayscale masks. This dataset serves to minimize the burdens of annotation and computational resources. The masks are annotated with 8-class labels, facilitating the training of neural networks for semantic segmentation tasks. A model containing a standard U-Net architecture with a ResNet-50 backbone, trained on this dataset performs with a mean intersection over union (meanIoU) score of 58 per cent. To improve the performance of our model, we employ pre-trained backbones. Utilizing a ResNet-50 backbone pre-trained on the 2012 ILSVRC ImageNet dataset elevates the meanIoU score to an impressive 71.7 per cent. Encouraged by such substantial improvements from a pre-trained backbone trained on natural images, we now propose the adoption of a ResNet-50 backbone, pre-trained through a self-supervised learning (SSL) method called MoCo, leveraging the SSL4EO-S12 dataset. This dataset contains extensive multimodal multitemporal unlabeled patch triplets, integrating Dual-pol SAR, S2-L1C TOA multispectral, and S2-L2A BOA multispectral data. Employing a ResNet-50 backbone solely pre-trained on one modality, S2-L2A RGB images from the SSL4EO-S12 dataset yield a striking meanIoU score of 77.5 per cent on our curated Chilean mining sites' dataset. It is worth noting that the remarkable performance boost achieved through SSL in our experimental setup underscores the significant value of SSL pre-training using specific remote sensing data/images, particularly in remote sensing segmentation tasks. This approach substantiates our initial objective of mitigating the adverse environmental impacts of mining operations.
A Texture and Saliency Enhanced Image Learning Method For Cross-Modal Remote Sensing Image-Text Retrieval

Cross-modal remote sensing image-text retrieval (CMRSITR) can retrieve images of interest from a vast amount of remote sensing images and has received significant attention in recent years. However, existing methods do not consider saliency and texture information, which are essential for remote sensing images when extracting image features. Therefore, this paper proposes a novel texture and saliency enhanced image learning method for CMRSITR. We constructed a multi-task image feature extractor in this new method. A texture map and a saliency map are created by extracting texture and detecting the saliency of each RS image. Both maps are set as supervised information during training to make the extracted saliency and texture features gradually reconstructed to a saliency map and a texture map, respectively. At the same time, the retrieval features of each RS image are obtained from the retrieval feature branch of the image. Experiments conducted on two commonly used CMRSITR datasets, RSICD and UCM, showed that the proposed method is effective in improving retrieval performance and achieved state-of-the-art retrieval performance compared to existing methods.
Have Foundational Models Seen Satellite Images?

This paper presents an investigation into the zero-shot performance of pre-trained foundation models on remote sensing tasks. Recent advances in self-supervised learning suggest that these models, when trained on vast amounts of unsupervised data, could potentially improve generalization across a number of downstream tasks. Our study offers an empirical evaluation of these models on standard remote-sensing benchmarks such as EuroSAT and BigEarthNet-S2, with the intent to confirm whether these models have encountered satellite imagery during their training phase. Moreover, we examine the impact of adding a geospatial domain-specific textual description of classes, contrasting it with the standard class-based prompts. Our findings indicate that the fine-tuned BLIP models exhibit superior zero-shot performance on these benchmarks compared to their standard counterparts, signifying that fine-tuning on standard benchmarks enhances performance. Furthermore, the addition of geospatial context variably influences performance depending on the specific model and dataset. This work provides crucial insights into the applicability of foundation models in remote sensing tasks and lays the groundwork for further research.