EXPLOITING SEMANTIC COHERENCE TO IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION

1 INTRODUCTION . Remote sensing scene image analysis is emerging as an important area of research for application of deep learning algorithms . Application areas include land-use land-cover analysis , urban planning , and natural disaster detection . A deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq . 1 Liu et al . ( 2019 ) , Albert et al . ( 2017 ) , Nogueira et al . ( 2016 ) , Castelluccio et al . ( 2015 ) , Mnih ( 2013 ) , Mnih & Hinton ( 2010 ) , P ( li|si ) ( 1 ) where li is label for image patch si . This formulation is sufficient for problems where spatial situatedness of a scene , which embodies knowledge of semantic likeness between neighborhoods in the geophysical world , is not important . However , for problems which require knowledge of neighborhood the formulation in Eq . 1 becomes inadequate . An example of such a problem would be estimating disease density for a small geographical region of interest , in which case the probability of label l is likely to depend on the labels for neighboring regions due to semantic coherence among them . The problem of how to improve model prediction by leveraging semantic coherence among neighboring scene images has previously been considered in the literature . Previous studies consider the problem as a post-classification task . For example , Bischof et al . ( 1992 ) used a second classifier to do pixel smoothing to refine predictions made by another classifier . Based on a 5x5 window , a filter assigns pixels to the majority class if it had been assigned a different class . In Mnih ( 2013 ) , a post-processing architecture is suggested for incorporating structure into image patch prediction . It involves stacking neural networks ( NN ) such that the output from a previous one becomes input for the next . Idea is for each network to clean up predictions of previous one in order to progressively improve overall accuracy . While improved model performance was achieved by these methods , they have overhead of performing same classification task in at least two stages . In other words , you need a minimum of two NN to perform the same classification task . Unlike post-classification methods , this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process . We make the assumption that l is conditionally co-dependent on information embedded in scene image i and in other similar , neighboring image j such that the problem is formulated as probability of the form in Eq . 2 , P ( li|Si , Sj ) = Πmi=1P ( li|si , sj ) ( 2 ) where sj is image for a neighboring tile that is most similar to index tile i andP ( li|Si , Sj ) is observed probability distribution . We used Convolutional Neural Networks ( CNN ) for modeling the observed probability distribution in Eq . 2 . A network architecture is proposed for training our model consisting of four components : a siamese sub-network , a similarity metric learning component , a convolutional network , and a decision layer . The siamese sub-network takes two neighboring scene images as input and extracts features from each . The similarity learning component evaluates how similar the input images are , using the extracted features . If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector , otherwise those from the index tile are used alone . We implemented the decision layer to perform classification or regression . A baseline model was implemented that takes a single image , the index tile i , as input . Empirical results show the proposed model consistently outperforms the baseline . In addition to improving predictive performance with a relatively small training set , our model is fast to train since it uses a pre-trained model for the siamese sub-network . Furthermore , it does not require another NN to smooth out its predictions as is the case with post-classification approaches , while achieving comparable performance gain . In summary , our contributions include the following . 1 . We propose an approach for training a probabilistic deep learning model to improve prediction accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images . A CNN architecture is suggested for this purpose . 2 . We provide empirical evidence that demonstrates the viability of this approach on a disease density estimation task . 3 . Lastly , we discovered an important limitation of the synthetic minority over-sampling technique ( SMOTE ) . This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be preserved , an important requirement under the framework of learning deep features over neighboring scene images introduced in this work . 2 RELATED WORK . Remote sensing scene images represent a difficult problem for deep learning algorithms due to high intra-class diversity and inter-class similarity ( Cheng et al. , 2017 ) . Previous efforts to address this problem is presented first , followed by recent use of siamese networks to improve discriminative ability of CNN . 2.1 IMPROVING MODEL ACCURACY BY EXPLOITING COHERENCE IN SCENE IMAGES . One of the earliest works aimed at improving model accuracy by exploiting semantic coherence between neighboring images is by Bischof et al . ( 1992 ) . The problem addressed there is ’ salt and pepper ’ noise in classified satellite images . The authors employ a multi-layer perceptron ( MLP ) consisting of five hidden units to classify low resolution ( 30m ) satellite scene images . Their proposed network has four output units corresponding to four land-use classes . The sigmoid activation function was used with both hidden and output units . The network had seven input units , each corresponding to one of seven Landsat TM spectral bands . It generates two separate channels when classifying an input image : one for class prediction result and the other representing how confident the classification is . A post-classification smoothing operation is then performed using a two-layer NN whose input is the classified image within a 5 x 5 window . For each pixel , the predicted class and confidence information are used as input to decide its new class belonging by applying a majority filter . Gain in classification accuracy was 2.9 percent for the network whose input data was enhanced with texture information ( and 4.9 percent for the one without enhancement ) . Mnih ( 2013 ) introduce indirect dependencies between outputs of a model by using knowledge of structure ( e.g . shape ) to resolve the issue of disconnected blobs and holes or gaps in predicted building and road network maps . The architecture used consists of stacking NN , each using as input the outputs of a previous network . Precisely , let M̂0 be the map predicted by a model . The ith level cleanup NN ( fi ) takes a ws x ws patch of ˆMi−1 and outputs a wm x wm patch of M̂i . The model fi is trained by minimizing the negative log likelihood on patches of the observed map M̂ , just like other NN in the architecture . Thus , the ith cleanup NN improves predictions of the previous NN fi−1 . The authors extended this idea to Conditional Random Fields ( CRF ) i.e . pairwise lattice CRF to introduce explicit dependencies between pairs of neighboring pixels . For both NN and CRF , an average gain of 0.0195 was achieved in precision and recall break-even points . The above works demonstrate that post-classification smoothing operations can improve performance of deep learning models . However , a draw back of those approaches is the necessary requirement to implement additional models to perform smoothing operations . Therefore , approaches that sidestep this requirement would be a welcome alternative . 2.2 SIAMESE NEURAL NETWORKS . Another innovation that inspired the current work is siamese neural networks ( SNN ) introduced by Bromley et al . ( 1993 ) and Baldi & Chauvin ( 1993 ) . A SNN architecture consists of two identical ( shared weight ) sub-networks merged by the same function at their outputs . During training each sub-network extracts features from one of two concurrent inputs , after which the joining function evaluates whether the input pair is similar based on a distance metric , such as cosine of angle between the pair of feature vectors . Output from the SNN is a decision score e.g. , 1 for similar image pair , 0 otherwise . SNN have recently been used for remote sensing scene image understanding . Liu et al . ( 2019 ) used a siamese architecture to improve discriminative ability of CNN for satellite scene image classification . The architecture consists of two identical CNN models , 3 additional convolutional layers , and one square non-parametric layer . Each branch of the siamese network is responsible for extracting features from each dual input image , giving feature vectors f1 and f2 . Two of the convolutional layers take these feature vectors as input and perform additional learning before a softmax function predicts the label for each input image . On the other hand , the square layer takes f1 and f2 as input and outputs the tensor fs = ( f1 − f2 ) 2 which is used to measure similarity between the input pair . The model thus produces three outputs : predicted label for each input image and a score for their similarity . More precisely , feature discrimination enhancement is achieved in the square layer by imposing a metric learning regularization term that minimizes Euclidean distance between a similar image pair and maximizes that for a dissimilar pair ( Eq . 3 ) , D ( xi , xj ) = ||f1 − f2||22 ( 3 ) where D ( xi , xj ) is distance between a pair of input images . A margin τ is set to separate as far apart as possible , the similar image pairs from dissimilar ones in feature space . If ( xi , xj ) is from the same scene class the distance between them is less than τ , otherwise it is greater than τ . Let label for training pair ( xi , xj ) be ( yi , yj ) , respectively . The formulation for Euclidean distance between them is given by ( Eq . 4 ) , { D ( xi , xj ) < τ , yi = yj D ( xi , xj ) > τ , yi 6= yj ( 4 ) This model only minimizes feature distance between similar input pairs through the margin τ to achieve feature discrimination . Therefore , the model in Liu et al . ( 2019 ) optimizes for two objective functions : a distance loss function and a regularization term as shown in Eq . 5 , Dist ( f1 , f2 ) = ∑ i , j D ( xi , xj ) , D ( xi , xj ) < τ ( 5 ) Of the three pre-trained models experimented with ( AlexNet , VGG-16 , and ResNet-50 ) highest feature discrimination accuracy gain ( 1.14 percentage point ) was achieved with a Siamese AlexNet model on the challenging NWPU-RESIC45 ( Cheng et al. , 2017 ) benchmark data set . Cheng et al . ( 2018 ) reported a 1.53 percentage point gain in accuracy on the same data set . 3 LEARNING DEEP FEATURES OVER NEIGHBORING SCENE IMAGES .

Summary: This papers proposed a solution to the problem of disease density estimation using satellite scene images. One common challenge in this type of applications is having a high intra-class diversity and a high inter-class similarity. The solution proposed by the authors is based on the use of siamese networks to extract features from pairs of neighbouring images, and merge the features only if they are similar. The authors claim that this approach alleviate the need of a post-classification smoothing.