A Domain-Change Approach to the Semantic Labelling of Remote Sensing Images
Authors: karmakar chandrabali , karmakar Gottfried , Dumitru Corneliu Octavian , Datcu Mihai 
Published: March 2022

&lt;p&gt;For many years, image classification &amp;#8211; mainly based on pixel brightness statistics &amp;#8211; has been among the&lt;br&gt;most popular remote sensing applications. However, during recent years, many users were more and&lt;br&gt;more interested in the application-oriented semantic labelling of remotely sensed image objects being&lt;br&gt;depicted in given images.&lt;br&gt;In parallel, the development of deep learning algorithms has led to several powerful image&lt;br&gt;classification and annotation tools that became popular in the remote sensing community. In most&lt;br&gt;cases, these publicly available tools combine efficient algorithms with expert knowledge and/or&lt;br&gt;external information ingested during an initial training phase, and we often encounter two alternative&lt;br&gt;types of deep learning approaches, namely Autoencoders (AEs) and Convolutional Neural Networks&lt;br&gt;(CNNs). Both approaches try to convert the pixel data of remote sensing images into semantic maps of&lt;br&gt;the imaged areas.&lt;br&gt;In our case, we made an attempt to provide an efficient new semantic annotation tool that helps in&lt;br&gt;the semantic interpretation of newly recorded images with known and/or possibly unknown content.&lt;br&gt;Typical cases are remote sensing images depicting unexpected and hitherto uncharted phenomena&lt;br&gt;such as flooding events or destroyed infrastructure. When we resort to the commonly applied AE or&lt;br&gt;CNN software packages we cannot expect that existing statistics, or a few initial ground-truth&lt;br&gt;annotations made by an image interpreter, will automatically lead to a perfect understanding of the&lt;br&gt;image content. Instead, we have to discover and combine a number of additional relationships that&lt;br&gt;define the actual content of a selected image and many of its characteristics.&lt;br&gt;Our approach consists of a two-stage domain-change approach where we first convert an image into&lt;br&gt;a purely mathematical &amp;#8216;topic representation&amp;#8217; initially introduced by Blei [1]. This representation&lt;br&gt;provides statistics-based topics that do not yet require final application-oriented labelling describing&lt;br&gt;physical categories or phenomena and support the idea of explainable machine learning [2]. Then,&lt;br&gt;during a second stage, we try to derive physical image content categories by exploiting a weighted&lt;br&gt;multi-level neural network approach that converts weighted topics into individual application-oriented&lt;br&gt;labels. This domain-changing learning stage limits label noise and is initially supported by an image&lt;br&gt;interpreter allowing the joint use of pixel statistics and expert knowledge [3]. The activity of the image&lt;br&gt;interpreter can be limited to a few image patches.&lt;br&gt;We tested our approach on a number of different use cases (e.g., polar ice, agriculture, natural&lt;br&gt;disasters) and found that our concept provides promising results.&amp;#160;&amp;#160;&lt;br&gt;[1] D.M. Blei, A.Y. Ng, and M.I. Jordan, (2003). Latent Dirichlet Allocation, Journal of Machine Learning&lt;br&gt;Research, Vol. 3, pp. 993-1022.&lt;br&gt;[2] C. Karmakar, C.O. Dumitru, G. Schwarz, and M. Datcu (2020). Feature-free explainable data mining&lt;br&gt;in SAR images using latent Dirichlet allocation, IEEE Journal of Selected Topics in Applied Earth&lt;br&gt;Observations and Remote Sensing, Vol. 14, pp. 676-689.&lt;br&gt;[3] C.O. Dumitru, G. Schwarz, and M. Datcu (2021). Semantic Labelling of Globally Distributed Urban&lt;br&gt;and Non-Urban Satellite Images Using High-Resolution SAR Data, IEEE Journal of Selected Topics in&lt;br&gt;Applied Earth Observations and Remote Sensing, Vol. 15, pp. 6009-6068.&lt;/p&gt;