EXPLOITING SEMANTIC COHERENCE TO IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION
Authors: Rahman Sanya
Gilbert Maiga
Ernest Mwebaze
Venue:  (2020)
Keywords: semantic coherence
satellite scene image analysis
convolutional neural networks
disease density

Abstract: High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks. To improve accuracy, post-classification
methods have been proposed for smoothing results of model predictions. However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model. Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair
of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label.
Empirical results show that this approach provides a viable alternative to existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with results from existing post-classification methods, moreover without implementation overheads.

Under review as a conference paper at ICLR 2020
EXPLOITING SEMANTIC COHERENCE To IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
High intra-class diversity and inter-class similarity is a characteristic remote sensing scene image data sets currently posing significant difficulty for deep learn- ing algorithms on classification tasks. To improve accuracy, post-classification methods have been proposed for smoothing results of model predictions. How- ever, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose approach that involves learning deep features directly over neighboring scene images with- out requiring of cleanup model. Our approach utilizes siamese network to improve the discriminative power of convolutional neural networks on a pair of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label. Empirical results show that this approach provides a viable alternative existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with re- sults from existing post-classification methods, moreover without implementation overheads.
INTRODUCTION
Remote sensing scene image analysis is emerging deep learning algorithms. Application areas include land-use land-cover analysis, urban planning, and natural disaster detection. deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq. Liu et al. (2019) Albert et al. (2017), Nogueira et al.
important area of research for application of
(2016), Castelluccio al. (2015) Mnih 2013). Mnih Hinton (2010).
P(([s.)
(1)
where 1; is label for image patch 8;. This formulation is sufficient for problems where spatial sit- uatedness of scene, which embodies knowledge of semantic likeness between neighborhoods in the geophysical world, is not important. However, for problems which require knowledge of neigh- borhood the formulation in Eq. [ becomes inadequate. An example of such problem would be estimating disease density for small geographical region of interest, in which case the probability of label is likely to depend on the labels for neighboring regions due semantic coherence among them.
The problem of how to improve model prediction by leveraging semantic coherence among neigh- boring scene images has previously been considered in the literature. Previous studies consider the problem as a post-classiffcation task. For example, Bischof et al. (1992) used a second classifier to do pixel smoothing to refine predictions made by another classifier. Based on 5x5 window, filter assigns pixels to the majority class if it had been assigned a different class. In Mnih (2013). a post- processing architecture is suggested for incorporating structure into image patch prediction. It involves stacking neural networks (NN) such that the output from a previous one becomes input for the next. Idea is for each network clean up predictions of previous in order to progressively

---

Under review as a conference paper at ICLR 2020
improve overall accuracy. While improved model performance was achieved by these methods, they have overhead of performing same classification task in at least two stages. In other words, you need minimum of two NN to perform the same classification task.
Unlike postccaassiiicttion methods, this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process. We make the assumption that / is conditionally co-dependent on information embedded in scene image and in other similar, neighboring image j such that the problem is formulated as probability of the form in Eq. 2
S,)) -11 P(4ss,
(2)
where s; is image for neighboring tile that is most similar to index tile i and P())), is observed probability distribution.
We used Convolutional Neural Networks (CNN) for modeling the observed probability distribution
in Eq. 2 A network architecture is proposed for training our model consisting of four components: siamese sub-network, similarity metric learning component, convolutional network, and decision layer. The siamese sub-network takes two neighboring scene images as input and extracts features from each. The similarity learning component evaluates how similar the input images are, using the extracted features. If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector, otherwise those from the index tile are used alone. We implemented the decision layer to perform classification or regression. baseline model was implemented that takes a single image, the index tile i, as input.
Empirical results show the proposed model consistently outperforms the baseline. In addition to improving predictive performance with a relatively small training set, our model is fast to train since it uses pre-trained model for the siamese sub-network. Furthermore, it does not require another NN to smooth out its predictions as is the case with post-classification approaches, while achieving comparable performance gain. In summary,our contributions include the following.
1. We propose an approach for training a probabilistic deep learning model to improve predic- tion accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images. A CNN architecture suggested for this purpose.
2. We provide empirical evidence that demonstrates the viability of this approach on discase density estimation task.
3. Lastly, we discovered an important limitation of the synthetic minority over-sampling tech- nique (SMOTE). This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be pre- served, an important requirement under the framework of learning deep features over neigh- boring scene images introduced in this work.
RELATED WORK
Remote sensing scene images represent a difficult problem for deep learning algorithms due high intra-class diversity and inter-class similarity (Cheng et al., 2017). Previous efforts to address this problem presented first, followed by recent use of siamese networks to improve discriminative ability of CNN.
2.1 IMPROVING MODEL ACCURACY BY EXPLOITING COHERENCE SCENE IMAGES
One of the earliest works aimed at improving model accuracy by exploiting semantic coherence between neighboring images is by Bischof et al. (1992). The problem addressed there 'salt and pepper' noise in classified satellite images. The authors employ multi-layer perceptron (MLP) consisting of five hidden units to classify low resolution (30m) satellite scene images. Their pro- posed network has four output units corresponding to four land-use classes. The sigmoid activation function was used with both hidden and output units. The network had seven input units, each cor- responding to one of seven Landsat TM spectral bands. generates two separate channels when
2

---

Under review as a conference paper at ICLR 2020
classifying an input image: one for class prediction result and the other representing how confident the classification is. A post-classification smoothing operation is then performed using a two-layer NN whose input is the classified image within 5 5 window. For each pixel, the predicted class and confidence information are used as input to decide its new class belonging by applying a ma- jority filter. Gain in classification accuracy was 2.9 percent for the network whose input data was enhanced with texture information (and 4.9 percent for the one without enhancement).
Mnih (0133) introduce indirect dependencies between outputs of model by using knowledge of structure (e.g- shape) to resolve the issue of disconnected blobs and holes or gaps in predicted building and road network maps. The architecture used consists of stacking NN, each using as input the outputs of a previous network. Precisely, let M, be the map predicted by model. The ith level cleanup NN (fi) takes a ws X w, patch of M and outputs a Wm X Wm patch of M.. The model fi is trained by minimizing the negative log likelihood on patches of the observed map M, just like other NN in the architecture. Thus, the ith cleanup NN improves predictions of the previous NN fi-1- The authors extended this idea to Conditional Random Fields (CRF) i.e. pairwise lattice CRF to introduce explicit dependencies between pairs of neighboring pixels. For both NN and CRF, an average gain of 0.0195 was achieved in precision and recall break-even points.
The above works demonstrate that post-classification smoothing operations can improve perfor- mance of deep learning models. However, a draw back of those approaches is the necessary require- ment to implement additional models to perform smoothing operations. Therefore, approaches that sidestep this requirement would be a welcome alternative.
2.2 SIAMESE NEURAL NETWORKS
Another innovation that inspired the current work is siamese neural networks (SNN) introduced by Bromley et al. (1993) and [Baldi & Chauvin (1993). A SNN architecture consists of two identical (shared weight) sub-networks merged by the same function at their outputs. During training each sub-network extracts features from one of two concurrent inputs, after which the joining function evaluates whether the input pair is similar based on distance metric, such as cosine of angle be- tween the pair of feature vectors. Output from the SNN is a decision score e.g., for similar image pair, O otherwise.
SNN have recently been used for remote sensing scene image understanding. Liu et al. (2019) used siamese architecture to improve discriminative ability of CNN for satellite scene image classifica- tion. The architecture consists of two identical CNN models, 3 additional convolutional layers, and one square non-parametric layer. Each branch of the siamese network is responsible for extracting features from each dual input image, giving feature vectors f1 and f2. Two of the convolutional layers take these feature vectors as input and perform additional learning before a softmax function predicts the label for each input image. On the other hand, the square layer takes fi and f2 as input and outputs the tensor fa = (f1 f22 which is used to measure similarity between the input pair. The model thus produces three outputs: predicted label for each input image and score for their similarity.
More precisely, feature discrimination enhancement is achieved in the square layer by imposing a metric learning regularization term that minimizes Euclidean distance between similar image pair and maximizes that for dissimilar pair (Eq.
D(E,, 11f1 fal12
(3)
where D(x,, x.)) is distance between pair of input images. A margin r is set to separate far apart as possible, the similar image pairs from dissimilar ones in feature space. If (I;, is from the same scene class the distance between them less than 7, otherwise is greater than r. Let label for training pair (I;, x;) ) be W;), respectively. The formulation for Euclidean distance between them is given by (Eq. 4).
{ D(ri, < T, = Mj ((z,, > T. + y)
(4)
3

---

Under review as a conference paper at ICLR 2020
This model only minimizes feature distance between similar input pairs through the margin to achieve feature discrimination.
Therefore, the model in Liu et al. (2019) optimizes for two objective functions: distance loss function and a regularization term as shown in Eq. S
fi, = D(),)))) D(2,, <T
(5)
Of the three pre-trained models experimented with (AlexNet, VGG-16, and ResNet-500 highest feature discrimination accuracy gain (1.14 percentage point) was achieved with Siamese AlexNet model on the challenging NWPU-RESIC45 (Cheng et al., 2017) benchmark data set. CCeng et al. 20188 reported a 1.53 percentage point gain in accuracy on the same data set.
LEARNING DEEP FEATURES OVER NEIGHBORING SCENE IMAGES
3.1 OVERVIEW
An end-to-end pipeline is proposed consisting of four components: siamese sub-network, similar- ity metric learning function, convolutional network, and decision layer. The siamese sub-network takes pair of scene images (tile want predict label for) and sj (other tile neighboring s;) as input and extracts features from each, outputting feature vectors fi and fj. respectively. The sim- ilarity function takes as input the feature vectors and learns a similarity metric between them. If the feature vector pair (f:, ff) is found to be similar they are merged and the resulting feature vector used train the convolutional network, otherwise training done using (fi) alone. Output from the convolutional network is passed as input to the decision layer to predict a label (or to regress numerical value) for image patch 8;. The proposed and baseline architectures are shown Figures la and lb respectively.
decidon tsyer
comoouatc natwrak
merge: 05) 65)
dm
sintarty venning finction
PHOKI ROO
(5)
(5)
Staeese sre-tnlded) model
Baseline architecture,
Tarnnigito igg
Fore
(a) Proposed network architecture.
Figure Network architectures used in this work.
3.2 ARCHITECTURE DESCRIPTION
3.2.1 SIAMESE SUB-NETWORK
To implement the siamese sub-network we explored two state-of-the-art pre-trained CNN models iec. Xception net Chollet (2066) and ResNet-50 He et al. (2016) in transfer learning (fine-tuning) strategy since we had a small training set. The architecture of Xception model based on separating learning of space- wise and channel-wise feature representations. The result is better performance
4

---

Under review as a conference paper at ICLR 2020
over architectures such as Inception V3, ResNet-152, and VGG-16 in terms of computational speed and classification accuracy. On the other hand, ResNet architectures address the problem of van- ishing/exploding gradients in deep neural network architectures by inserting shortcut connections between convolutional blocks. The ResNet model has benefit of enabling deeper architectures to be utilized in CNN models since these provide higher accuracy rates than shallow architectures.
3.2.2 SIMILARITY METRIC LEARNING
The similarity function takes as input feature vector (fs: ff) to evaluate how similar the input image pair is. This unit outputs one of a binary decision score i.e neighboring input pair (s;, ) similar (1) or dissimilar (0). The optimization function used here works by minimizing feature distance between similar images. Following work in Liu et al. (2019), given a pair of input image 8; and 8j the distance between them is calculated using Eq.
A margin separates similar image features from dissimilar ones such that D(ft, fy) < T for a similar pair, D(f::)) > T otherwise. We also optimized for two objective functions defined in Eq. [
An important question to consider is how to identify a neighboring image that is semantically closest in likeness to index tile i, over which to co-leam features. For any image patch in the middle region of a w x scene image S where w23 and h23, there will be eight such tiles to choose from, considering four adjacent and diagonal neighbors. For edge tiles there will be five while for corner tiles there will be three neighbors, Figure 2 in Appendix.
We determine neighhors of the tile location by specifying the maximum Harversine distance within which the centroid coordinates of all neighbors will lie. Take as an example tiles of size 250m² x 3 scene image. The centroid of all eight neighbors of the center tile would fall within a maximum distance of 350m from its own centroid. We can evaluate how similar tile at is to its neighbor at location using Euclidean distance. If at least one is similar, we select the neighbor with least distance to be most similar. Given that (ss 8j) < T for images from same scene class, a neighboring image is most similar to 8; if it has the smallest distance Eq. (6]
= arg min(si, 83)
where &; is neighboring image most similar index tile at location i.
In the current analysis however, we consider only the preceding neighbor to tile at location i (image s(-1) for similarity analysis. The loss function used is binary cross-entropy, defined in Eq.
L(W,b) = Σ ==1
x log + (1 - vi) x log (1 - Di)
3.2.3 CONVOLUTIONAL NETWORK
This network is trained using features resulting from merging feature vector pair (fr, fz) if respective input image pair (s,, sy) is found to be similar and it outputs feature vector (m fj). If the input pair is dissimilar, only the feature vector f; is fed to the convolutional network, which outputs f.(..) Our implementation used three dense layers in the convolutional network. Merge operation used is feature averaging.
3.2.4 DECISION LAYER
For classification task we learn the parameters (weights) of the CNN by minimizing the negative log likelihood of the training data under our model. For the multi-class problem considered in this work the negative log likelihood under the model Eq. [ assumes the form of a cross-entropy between the probability distribution for observed labels I and the predicted label probabilities ý (Eq.
8).
L(W,b)
Σ
Σ
ys log y
(8)
1
5

---

Under review as a conference paper at ICLR 2020
where M. the number of classes, is 3.
The outer sum of objective function is over all training samples. Stochastic gradient descent with mini-batches is used for optimizing L. Our implementation for the current problem uses fully connected network (FCN) with a 3-way softmax classifier.
Taken regression task, optimized for the mean absolute error (MAE) loss and mean squared error (MSE). The form of MAE we used given by Eq. 9
N
M
L(W,b)
2N
j=1
where M. the number of features be predicted, For MSE the loss function given by Eq. 10
M
L(W. b)
1
(
y;)²
(10)
N
M
EXPERIMENTS
We applied our method the task of estimating disease density from satellite scene imagery. Below we describe data sets and methods used.
4.1 METHOOS AND DATASET
Dataset
number of data sets were used in our experiments. The epidemic data consists monthly disease case counts aggregated by sub-county for year 2015. We dis-aggregated the data to 250m grids, our geospatial unit of analysis, using population data from Facebook & CIESIN (2016). The latter data gives estimate of people living in a 30m² grid based recent census data and high resolution (0.5m²) satellite imagery. The data was up-sampled to 250m² grid to consistent with our spatial unit of analysis. The population data epidemic data. The building (housing) concentration data consists of satellite imagery extracted from Google Static Maps API using method in Sanya & Mwebazel (2018). It consists of land-use type buildings i.e. objects of interest in a satellite scene image is buildings or houses. The housing data was used as input to train the model, as proxy for indoor overcrowding
used to create weighting scheme for dis tis-aggregating
known risk factor for our case study disease. All imput data was pre- rrepprocsseed into feature vectors and normalized to (0,1) value range by min-max scaling or (-1,1) by standardizing (zero mean, unit variance).
Methods
We created disease density classes out of the epidemic data make it a classification task. We did this by binning the normalized data such that each 250m² grid assumed class value depending on which bin its disease density lies in. Following the procedure in Robinson al. (2017) for binning population data, we created matrix C where an entry C; 0 if 5.74e 05 d: 1.52e 4, 1 if 1.52e < d, < 2.46e 4, and 2 if 2.46e - < d; < 3.4e 4, where d, is normalized disease density. The classes correspond to the semantic labels 0. 1. 2 for high, low, and moderate disease density with 3,628, 6,472, and 1,970 membership count, respectively.
Let D be grid of disease case counts for region of interest at time t, C' grid of disease density class label values, and S grid of housing concentration data. For every disease case count value d, and density class label value c; there is associated housing concentration image s,.
We formulated the learning task as estimating unknown function f based on the idea of learning deep features over neighboring scene images (one neighbor in this case) using Eq. 11 (Figure) Ta
(5()))8)) if D(si, si) < and arg min., (D(si, 8;)), (((s)
(11)
if D(s,, 8,) > T.

---

Under review as a conference paper at ICLR 2020
Table 1: Overall accuracy, precision, recall, and Fl-score for similarity metric learning
Accuracy
Precision
Recall
ResNet-50 57
0.39 0.70
0.10
Xception 83
0.60
Table 2: Overall accuracy, precision, recall, F1-score for baseline and proposed model.
Accuracy
Precision
Recall
F1-score
Bline
Model
Bline
Model
Bline
Model
Bline
Model
ResNet-50 44
42
0.0
0.11
0.0
0.3
0.0
0.5
Xception
33
34
0.13
0.15
0.26
0.39
0.18
0.22
where c; estimated disease density for a 250mm² patch of land on earth's surface (approx. 224m² pixels for image), represented by image s;. sj image of neighboring scene that most similar to image of tile at location i, identified using method in Eq.
We used CNN estimate function f because the mapping from input data disease incidence estimate is non-linear, noisy, and dependent on semantic content of input data. To train our model we used transfer learning by fine-tuning two pre-trate pre-trained models on our data set i.c., Xception and ResNet-50. Transfer learning eliminates network architecture design time while often achieving higher accuracy rates than training model from scratch.
We applied the same regularization methods and other hyperparameter values (Table Appendix) to ensure uniformity across the proposed and baseline network architectures. Once a satisfactory model was achieved, it was trained on combined data set (12,070 images) of training (9,656) and validation (2,414) before evaluating on the test set (1,189).
4.2
RESULTS
4.2.1
SIMILARITY METRIC LEARNING
Evaluation results for similarity metric learning for our proposed model built on top of ResNet-50 and Xception are shown in Table I ResNet-50 performed better than Xception net on precision (0.39 vs. 0.70) and recall (0.10 vs. 0.60) on the task of detecting whether or not a pair of satellite scene images is similar, even though the overall performance is low for both pre-traines models (maximum OA score of 83 percent got with Xception net).
4.2.2 CLASSIFICATION PERFORMANCE
Performance results for the baseline and proposed model are shown in Table 2 Overall, the Xcep- tion based model performed better than the ResNer-50 based one the task of estimating disease density from housing satellite scene data when performance is measured using precision, recall, and FI-Score metrics. However, both models generally performed poorly, achieving maximum overall accuracy score of 44 percent for ResNet-50. Our proposed model however, consistently out performed the baseline across both pre-trained models (scores marked with bold). For example, model achieved precision score of 0.15 0.13,
using Xception net the Siamese network recall of 0.39 vs. 0.26, and fl-score of 0.22 vs. 0.18 for the baseline model, respectively. similar performance trend is observed for the model based on ResNet-50.
Confusion matrix plots for baseline and our model built using Xception net are shown in Figure 3a and 3b] respectively (Appendix). On the other hand Figure 4 shows AUROC plots. Both show that our model performed better than the baseline on medium and low disease density classes with 58 48 percent and 47 37 percent, respectively. However, the model performed worse than the

---

Under review as a conference paper at ICLR 2020
baseline high disease density class (0 vs. 13 percent, respectively). Generally, the results from both models are as good as chance.
4.2.3 REGRESSION PERFORMANCE
Table MAE and MSE scores for baseline and proposed model. Overall, the model built using ResNet-50 performed better than the built with Xception net when used estimate disease den- sity from housing concentration satellite scene image data. Again, our proposed model performed consistently (even though marginally) better than the baseline (scores marked with bold). achieved MAE score of 0.35 vs. 0.38 and MSE score of 0.18 vs. 0.20 for the baseline, respectively. A similar trend is observed for the model built with Xception net as base.
Table 3: MAE and MSE scores for baseline and proposed model.
MAE
MSE
Baseline
Model
Baseline
Model
ResNet-50
0.38 0.39
0.35 0.37
0.20 0.20
0.18
Xception
0.19
5 DISCUSSION
Our model performed better than the baseline in both the classification and regression tasks for dis- ease density estimation. For example, model achieved I percentage point gain in accuracy the baseline model. While a gain as result of deploying a siamese network to boost discriminative power of CNN for aerial scene image classification is consistent with findings in previous studies overall results from our model are poor. For instance, our model was only able to attain a maximum overall accuracy of 34 percent on the classification task. We would like to pin these poor results to combination of three factors. First, the small data set used to train our model (12,070 images) could have impacted accuracy negatively, despite the use of regularization methods.Ir It is therefore, possible that our model suffered the problem of overfitting. Secondly, our data set was unbalanced. likely that the extra parameter we introduced in the loss function to weight classes by giving higher importance to under-represented classes did not work as expected. The result could have been that our model did not learn all the necessary features required to make a prediction but rather could have resorted to guessing the output and hence, failing to generalize well over the test set. Class imbalance in our data set could also have negatively affected feature correlation which in turn could have reduced model performance. Besides, well-known methods for mitigating sample size bias in imbalanced data sets, for example by over-sampling under-represented classes Chawla la et al. (2002) could be applied directly to our data set without modifying the algorithm. That is because was not immediately clear how to preserve spatial proximity between neighboring tiles, an idea that is central learning deep features over neighboring scene images. Howevvr, despite the low overall performance by our model we have been able to demonstrate that it is possible to improve model accuracy by learning deep features over neighboring scene images disease density estimation task.
REFERENCES
Adrian Albert, Jasleen Kaur, and Marta C. Gonzalez. Using convolutional networks and satellite im- agery to identify patterns in urban environments at large scale. ACM SigKDD 2017 Conference, 2017.
Pierre Baldi and Yves Chauvin. Neural networks for fingerprint recognition. Neural Comput., 5(3):402 418, May 1993. ISSN 0899-7667. doi: 10. 00.116///co.11993. 1993.5.3.402.
URL http:
7/dx. doi orgg 10. n 1993. 4.42
8

---

Under review as a conference paper at ICLR 2020
Horst Bischof, Wolfger Schneider, and Axel J. Pinz. Multispectral classification of landsat-imager using neural networks. IEEE Transactions on Geoscience and Remote Sensing, 30(3):482 490, May 1992. ISSN 0196-2892. doi: 1.11119.61992222
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a "siamese" time delay neural network. In Proceedings of the 6th International Con- ference on Neural Information Processing Systems, NIPS 93, pp. 737-744, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. URL http: d1 acm org.c/aaa.iaiii...........
cfm? ftmmidd2287180 89.
Marco Castelluccio, Giovanni Poggi, Carlo Sansone, and Luisa Verdoliva. Land use classification in remote sensing images by convolutional neural networks. CoRR, 2015.
Nitech V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and Philip W. Kegelmeyer. Smote: Syn- thetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002:321- 357, 2002.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state-of-the-eeee Proceedings of the IEEE, 105(10):18635 865-1883, 2017.
Gong Cheng, Ceyuan Yang. Xiwen Yao, Lei Guo, and Junwei Han. When deep learning meets met- ric learning: Remote sensing image scene classification via learning discriminative cnns. IEEE Transactions Geoscience and Remote Sensing, 56(5):2811 2821, May 2018. ISSN 0196-2892.
doi: 10.1009/TRSS2201223399..
François Chollet. Xception: Deep learning with depthwise separable convolutions.
CoRR,
abs/1610.02357,, 2016.
htapg//bb//15/ 02357
Facebook and CIESIN. High resolution settlement layer (hrsl). source imagery for hrsl 2016 digi- talglobe, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. in 2016 ieee conference on computer vision and pattern recognition. CVPR, pp. 770778, 2016.
Xuning Liu, Yong Zhou, Jiaqi Zhao, Rui Yao, Bng Liu, and Yi Zheng. Siamese convolutional neural networks for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, pp. 1-5, 2019. ISSN 1545-598X. doi: 10. G0.1 2899299999.
Volodymyr Mnih. Machine learning for aerial image labeling. PhD thesis, University Toronto., 2013.
Volodymyr Mnih and Geofrey Hinton. Learning detect roads in high-resolution aerial images. In Daniilidis K, Maragos P, and Paragios (eds.), Proceedings of the 11th European Conference on Computer Vision (ECCV), 2010. doi: https://doi.org/10. 10071/78--3-44-111111 1--442--5567-3116
Keiller Nogueira, Otávio Augusto Bizetto Penatti, and Jefersson Alex dos Santos. Towards bet- ter exploiting convolutional neural networks for remote sensing scene classification. CoRR, abs/1602.00517, 2016.
URL http: wwwp:iiivorr..aa.ee.. www.b//11.... 01517
Caleb Robinson, Fred Hohman, and Bistra Dilkina. A deep learning approach for population es- timation from satellite imagery. CoRR, abs/ hos/1708.09080 2017. URL
Rahman Sanya and Ernest Mwebaze. Mapping spatial housing patterns using deep neural networks and remote sensing data. In Neural Information Processing Systems Conference 2018 Machine Learning for the Developing World Workshop. 2018
APPENDIX
9

---

Under review as a conference paper at ICLR 2020
Figure 2: Finding neighboring image j that semantically most similar image i.
Table 4: Hyperparameter values used in our experiments
Hyperparameter
Method or Value
Activation function (convolutional layer)
ReLU
Activation function (classification layer)
Softmax
Optimization method
SGD (Adadelta)
Loss function
Multi-class cross-entropy
Learning rate
0.1 (halved every 10 epochs)
Batch normalization
(momentum=0.9)
Weight initialization
Mean=0, std=0.01
Bias initialization
Mean=0.5, std=0.01
Weight decay type (kernel regularizer)
L2 (0.01)
Amount of weight decay
0.0002
Drop out Mini-batch size
Not used
16
Number of epochs
50
1.0
1.0
0.373684
0,200000
0662536
low
04489477
0.188571
0.000000
0.0
0.8
1.052632
0.6
1.136842
0.885711
0.000000
0.6
0.4
high
1.131599
0.33333
0.329111
High
1.126316
0.485238
0.000000
redium
mediumm
Predicted label
Predicted label
(a) Confusion matrix plot for baseline.
(b) Confusion matrix plot for our model.
Figure Confusion matrix plots for models used in this study.
08
0.
06
06
04-
Feoro Jenrage cure pone
04-
ricro average curve (one
hracr-average
macre-averae
02
02
ROC curve clsa 2 (area - 0.4)
ROC curve clus 2 (area - 0.50)
02
04
06
da
16
02
06
da
(a) AUROC plot for baseline.
(b) AUROC plot for our model.
Figure 4: AUROC plots for models used in this study.
10


Reviews:
[{"review_id": "rklgjG6Zjr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This papers proposed a solution to the problem of disease density estimation using satellite scene images. One common challenge in this type of applications is having a high intra-class diversity and a high inter-class similarity. The solution proposed by the authors is based on the use of siamese networks to extract features from pairs of neighbouring images, and merge the features only if they are similar. The authors claim that this approach alleviate the need of a post-classification smoothing.  \n\nAdvantages: \nThe idea of merging siamese features for similar tiles only is sound. The paper is clearly written and structured. The shown results seem to outperform the baseline. \n\nDrawbacks:\nThe paper seems of fairly limited novelty. Moreover, it is centered around  one particular application. Although the task is approached with both a classification and a regression model, the classification dataset is obtained by a simple binning which makes the two tasks highly related. It would be interesting to have different settings to test the consistency of improvement with the proposed method. Finally, the authors claim that the method alleviates the need to post-classification smoothing, but this cannot be straightforwardly concluded from the conducted experiments. It would be interesting to have a more thorough comparison to other methods that use post-classification processing. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HylZ2Rw6FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to extract features utilizing the adjacency between patches, for better classification/regression of satellite image patches. The proposed method achieves better results compared to a straightforward baseline method.\n\n\nI have several significant concerns:\n\n- In the abstract, the authors claim that existing approaches such as post-classification add computational overhead to the task, whereas the proposed method does not add significant overhead. However, to me, post-classification can be very simple and straightforward, whereas the proposed method adds a series of computations: the proposed method not only extracts features from the input image, but also for another neighboring image; then features are combined (if two images are similar), before feeding into the network. The authors need to validate the claim that their method is more efficient.\n\n- The baseline the authors compare to is weak. There are existing works on satellite image classification/regression. Many of them also use semantic/contextual information, or aim to improve the robustness of features. For example:\n\n[1] Derksen et al. Spatially Precise Contextual Features Based on Superpixel Neighborhoods for Land Cover Mapping with High Resolution Satellite Image Time Series. IGARSS 2018.\n\n[2] Ghassemi et al. Learning and Adapting Robust Features for Satellite Image Segmentation on Heterogeneous Data Sets. Geoscience and Remote Sensing 2019.\n\nI understand that the authors cannot compare to everything. But the authors should compare to representative baseline methods. Methods mentioned in the related work section (Section 2.1) can also be compared to.\n\n- The proposed method is very application specific. The author only discussed the remote sensing application. Given the ICLR community's interest in general methods that can be applied to (or already been tested on) multiple applications, the paper would have been stronger if the methods applicabilityto other domains was discussed (and even better demonstrated).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByewhdO65r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to do a coupled inference over pairs of geographically close images instead of a single image for satellite imagery. The coupling is done with an average pooling of the feature vectors when the neighbouring patches are detected to be similar enough based on a threshold on the L2 distance of these features. The method is applied to tasks of estimating crowding population, and diseases density, from satellite images. \n\nThe paper have little novelty. The approach reduces to a smoothing method over pairs of neighbouring patches, that is only activated sometimes based on a hard threshold. This seems arbitrary and there are many competing approaches that could be applied. \nOne could think about taking all the geograpical neighbourhood of a patch into account when making a prediction, e.g. with a coarse-to-fine prediction approach; the aggregation of features can be learned and more sophisticated than average pooling. Using a single-image baseline is not fair. The discussion is not up to the level of ICLR and offers mostly guesswork.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers proposed a solution to the problem of disease density estimation using satellite scene images.  The method combines a classification and regression task.  The reviewers were unanimous in their recommendation that the submission not be accepted to ICLR.  The main concern was a lack of methodological novelty.  The authors responded to reviewer comments, and indicated a list of improvements that still remain to be done indicating that the paper should at least go through another review cycle.
Decision: Reject