EXPLOITING SEMANTIC COHERENCE TO IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION

1 INTRODUCTION . Remote sensing scene image analysis is emerging as an important area of research for application of deep learning algorithms . Application areas include land-use land-cover analysis , urban planning , and natural disaster detection . A deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq . 1 Liu et al . ( 2019 ) , Albert et al . ( 2017 ) , Nogueira et al . ( 2016 ) , Castelluccio et al . ( 2015 ) , Mnih ( 2013 ) , Mnih & Hinton ( 2010 ) , P ( li|si ) ( 1 ) where li is label for image patch si . This formulation is sufficient for problems where spatial situatedness of a scene , which embodies knowledge of semantic likeness between neighborhoods in the geophysical world , is not important . However , for problems which require knowledge of neighborhood the formulation in Eq . 1 becomes inadequate . An example of such a problem would be estimating disease density for a small geographical region of interest , in which case the probability of label l is likely to depend on the labels for neighboring regions due to semantic coherence among them . The problem of how to improve model prediction by leveraging semantic coherence among neighboring scene images has previously been considered in the literature . Previous studies consider the problem as a post-classification task . For example , Bischof et al . ( 1992 ) used a second classifier to do pixel smoothing to refine predictions made by another classifier . Based on a 5x5 window , a filter assigns pixels to the majority class if it had been assigned a different class . In Mnih ( 2013 ) , a post-processing architecture is suggested for incorporating structure into image patch prediction . It involves stacking neural networks ( NN ) such that the output from a previous one becomes input for the next . Idea is for each network to clean up predictions of previous one in order to progressively improve overall accuracy . While improved model performance was achieved by these methods , they have overhead of performing same classification task in at least two stages . In other words , you need a minimum of two NN to perform the same classification task . Unlike post-classification methods , this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process . We make the assumption that l is conditionally co-dependent on information embedded in scene image i and in other similar , neighboring image j such that the problem is formulated as probability of the form in Eq . 2 , P ( li|Si , Sj ) = Πmi=1P ( li|si , sj ) ( 2 ) where sj is image for a neighboring tile that is most similar to index tile i andP ( li|Si , Sj ) is observed probability distribution . We used Convolutional Neural Networks ( CNN ) for modeling the observed probability distribution in Eq . 2 . A network architecture is proposed for training our model consisting of four components : a siamese sub-network , a similarity metric learning component , a convolutional network , and a decision layer . The siamese sub-network takes two neighboring scene images as input and extracts features from each . The similarity learning component evaluates how similar the input images are , using the extracted features . If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector , otherwise those from the index tile are used alone . We implemented the decision layer to perform classification or regression . A baseline model was implemented that takes a single image , the index tile i , as input . Empirical results show the proposed model consistently outperforms the baseline . In addition to improving predictive performance with a relatively small training set , our model is fast to train since it uses a pre-trained model for the siamese sub-network . Furthermore , it does not require another NN to smooth out its predictions as is the case with post-classification approaches , while achieving comparable performance gain . In summary , our contributions include the following . 1 . We propose an approach for training a probabilistic deep learning model to improve prediction accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images . A CNN architecture is suggested for this purpose . 2 . We provide empirical evidence that demonstrates the viability of this approach on a disease density estimation task . 3 . Lastly , we discovered an important limitation of the synthetic minority over-sampling technique ( SMOTE ) . This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be preserved , an important requirement under the framework of learning deep features over neighboring scene images introduced in this work . 2 RELATED WORK . Remote sensing scene images represent a difficult problem for deep learning algorithms due to high intra-class diversity and inter-class similarity ( Cheng et al. , 2017 ) . Previous efforts to address this problem is presented first , followed by recent use of siamese networks to improve discriminative ability of CNN . 2.1 IMPROVING MODEL ACCURACY BY EXPLOITING COHERENCE IN SCENE IMAGES . One of the earliest works aimed at improving model accuracy by exploiting semantic coherence between neighboring images is by Bischof et al . ( 1992 ) . The problem addressed there is ’ salt and pepper ’ noise in classified satellite images . The authors employ a multi-layer perceptron ( MLP ) consisting of five hidden units to classify low resolution ( 30m ) satellite scene images . Their proposed network has four output units corresponding to four land-use classes . The sigmoid activation function was used with both hidden and output units . The network had seven input units , each corresponding to one of seven Landsat TM spectral bands . It generates two separate channels when classifying an input image : one for class prediction result and the other representing how confident the classification is . A post-classification smoothing operation is then performed using a two-layer NN whose input is the classified image within a 5 x 5 window . For each pixel , the predicted class and confidence information are used as input to decide its new class belonging by applying a majority filter . Gain in classification accuracy was 2.9 percent for the network whose input data was enhanced with texture information ( and 4.9 percent for the one without enhancement ) . Mnih ( 2013 ) introduce indirect dependencies between outputs of a model by using knowledge of structure ( e.g . shape ) to resolve the issue of disconnected blobs and holes or gaps in predicted building and road network maps . The architecture used consists of stacking NN , each using as input the outputs of a previous network . Precisely , let M̂0 be the map predicted by a model . The ith level cleanup NN ( fi ) takes a ws x ws patch of ˆMi−1 and outputs a wm x wm patch of M̂i . The model fi is trained by minimizing the negative log likelihood on patches of the observed map M̂ , just like other NN in the architecture . Thus , the ith cleanup NN improves predictions of the previous NN fi−1 . The authors extended this idea to Conditional Random Fields ( CRF ) i.e . pairwise lattice CRF to introduce explicit dependencies between pairs of neighboring pixels . For both NN and CRF , an average gain of 0.0195 was achieved in precision and recall break-even points . The above works demonstrate that post-classification smoothing operations can improve performance of deep learning models . However , a draw back of those approaches is the necessary requirement to implement additional models to perform smoothing operations . Therefore , approaches that sidestep this requirement would be a welcome alternative . 2.2 SIAMESE NEURAL NETWORKS . Another innovation that inspired the current work is siamese neural networks ( SNN ) introduced by Bromley et al . ( 1993 ) and Baldi & Chauvin ( 1993 ) . A SNN architecture consists of two identical ( shared weight ) sub-networks merged by the same function at their outputs . During training each sub-network extracts features from one of two concurrent inputs , after which the joining function evaluates whether the input pair is similar based on a distance metric , such as cosine of angle between the pair of feature vectors . Output from the SNN is a decision score e.g. , 1 for similar image pair , 0 otherwise . SNN have recently been used for remote sensing scene image understanding . Liu et al . ( 2019 ) used a siamese architecture to improve discriminative ability of CNN for satellite scene image classification . The architecture consists of two identical CNN models , 3 additional convolutional layers , and one square non-parametric layer . Each branch of the siamese network is responsible for extracting features from each dual input image , giving feature vectors f1 and f2 . Two of the convolutional layers take these feature vectors as input and perform additional learning before a softmax function predicts the label for each input image . On the other hand , the square layer takes f1 and f2 as input and outputs the tensor fs = ( f1 − f2 ) 2 which is used to measure similarity between the input pair . The model thus produces three outputs : predicted label for each input image and a score for their similarity . More precisely , feature discrimination enhancement is achieved in the square layer by imposing a metric learning regularization term that minimizes Euclidean distance between a similar image pair and maximizes that for a dissimilar pair ( Eq . 3 ) , D ( xi , xj ) = ||f1 − f2||22 ( 3 ) where D ( xi , xj ) is distance between a pair of input images . A margin τ is set to separate as far apart as possible , the similar image pairs from dissimilar ones in feature space . If ( xi , xj ) is from the same scene class the distance between them is less than τ , otherwise it is greater than τ . Let label for training pair ( xi , xj ) be ( yi , yj ) , respectively . The formulation for Euclidean distance between them is given by ( Eq . 4 ) , { D ( xi , xj ) < τ , yi = yj D ( xi , xj ) > τ , yi 6= yj ( 4 ) This model only minimizes feature distance between similar input pairs through the margin τ to achieve feature discrimination . Therefore , the model in Liu et al . ( 2019 ) optimizes for two objective functions : a distance loss function and a regularization term as shown in Eq . 5 , Dist ( f1 , f2 ) = ∑ i , j D ( xi , xj ) , D ( xi , xj ) < τ ( 5 ) Of the three pre-trained models experimented with ( AlexNet , VGG-16 , and ResNet-50 ) highest feature discrimination accuracy gain ( 1.14 percentage point ) was achieved with a Siamese AlexNet model on the challenging NWPU-RESIC45 ( Cheng et al. , 2017 ) benchmark data set . Cheng et al . ( 2018 ) reported a 1.53 percentage point gain in accuracy on the same data set . 3 LEARNING DEEP FEATURES OVER NEIGHBORING SCENE IMAGES .

Summary: The paper proposes to do a coupled inference over pairs of geographically close images instead of a single image for satellite imagery. The coupling is done with an average pooling of the feature vectors when the neighbouring patches are detected to be similar enough based on a threshold on the L2 distance of these features. The method is applied to tasks of estimating crowding population, and diseases density, from satellite images.