Detecting Deforestation from Satellite Imagery | by Nick Condo | Towards Data Science

For my Capstone Project as part of Udacity’s Machine Learning Engineer Nanodegree, I was tasked with solving a problem of my choosing by applying machine learning algorithms and techniques learned throughout the course. I decided to choose an area which I am adamantly passionate about: saving the environment. I learned of a past Kaggle competition which involved classifying satellite images of the Amazon rainforest through the fantastic fast.ai course. You can find the competition here. I decided to embark on my journey of detecting deforestation of the rainforest, and see just how far up the Kaggle leaderboard I could climb.
The Amazon is the largest and most biodiverse tropical rainforest in the world, covering an area of 2.1 million square miles. It is comprised of an estimated 390 billion individual trees divided into 16,000 species. The Amazon has been referred to as the “lungs of the planet”, as it helps to stabilize the earth’s climate and slow global warming by fixing CO2 and producing 20% of the world’s oxygen.
Since 1978 over 289,000 square miles of Amazon rainforest have been destroyed across Brazil, Peru, Columbia, Bolivia, Venezuela, Suriname, Guyana, and French Guiana. Every minute, the world loses an area of forest the size of 48 football fields. There is concern that the destruction of the forest will result in loss of biodiversity, habitat loss, and the release of the carbon contained within the vegetation, which could accelerate global warming. Better data about the location of deforestation and human encroachment on forests can help governments and local stakeholders respond more quickly and effectively.
The goal of this project is to track changes in the Amazon rainforest due to deforestation using satellite image data. The data has been provided by Planet and hosted on Kaggle as part of a previous competition — Planet: Understanding the Amazon from Space. The labels for this task were chosen in collaboration with Planet’s Impact team and represent a reasonable subset of phenomena of interest in the Amazon basin. The task is a multi-label image classification problem, where each image will have one and potentially more than one atmospheric label and zero or more common and rare labels.
Models are evaluated based on their mean F2 score. The F score, commonly used in information retrieval, measures accuracy using the precision p and recall r. Precision is the ratio of true positives (tp) to all predicted positives (tp + fp). Recall is the ratio of true positives to all actual positives (tp + fn). The F2 score is given by:
For this task, I chose to implement deep learning models using the fastai library, which simplifies training neural nets using modern best practices. One of the staples of the fastai library is its use of cyclical learning rates. This is a relatively unknown technique (maybe not true any longer due to the popularity of the fast.ai courses) of setting the learning rate, which improves classification accuracy while needing fewer iterations vs fixed learning rate values. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Another benefit of fastai is that it makes transfer learning incredibly easy:
from fastai.vision import *learn = cnn_learner(data, models.resnet50)
My final solution consists of an ensemble of five models all pre-trained on ImageNet: resnet50, resnet101, resnet152, densenet121, and densenet169. I fine-tuned each model separately, splitting the training data into 80% training and 20% validation. I used progressive resizing of the images to train the models, a technique I learned about from the fast.ai course, taught by Jeremy Howard. Progressive resizing is a method of training computer vision models where you first train the model on a smaller size than the original, then continue to increase the size of the image two or three times until you finally train on the original image size. I first trained the models on 64x64 pixel images for five epochs with only the last layer of the network trainable, then another five epochs with the entire network trainable. I do the same for 128x128 pixel images, and then the original 256x256 pixel images. Finally, I trained for five more epochs on all of the available training data, combining the training and validation sets. I also tried semi-supervised learning, using the labels from my best submission to train on the test set, but found my performance to slightly decrease. I use batch sizes of 64 during the entire training process, while the learning rates are chosen prior to each block of five epochs using fastai’s built-in function for finding the best learning rate:
learn.lr_find()learn.recorder.plot()
In order to get a baseline score with which to measure future experiments’ impact, I implemented a resnet50 without progressive resizing, using an image size of 256x256 pixels. I did not use test-time augmentation, and set the threshold to 0.20. This model achieved an F2 score of 0.92746 on the competition’s private leaderboard. To improve upon the baseline score, I first implemented progressive refinement. I experimented with a couple different sizes, and settled on 64 -> 128 -> 256, which bumped up my F2 score to 0.92881. After utilizing test-time augmentation my score increased further to 0.92979. I then began experimenting with thresholding the predictions to see if I could squeeze out a couple more tenths of a percent. I was able to increase my score to 0.93024 by setting the threshold to 0.17. Finally, I combined the training and validation sets and trained for five more epochs to achieve a score of 0.93041 with the single resnet50 model.
I followed the same process as above with the other four models, and by creating an ensemble of the five models I was able to achieve an F2 score of 0.93173 on the private leaderboard. This score would’ve been good for 19th place out of 938 teams in the competition, only 0.00144 behind 1st place.
One of the techniques I would like to have tried but did not have the time/resources is Single Image Haze Removal Using Dark Channel Prior — He, et al. I believe this would’ve been quite beneficial as a many of the satellite images included haze and/or clouds which made it more difficult for the model to classify the underlying features of the forest. In fact, the winner of the competition mentioned that he used this technique in his brief description of his solution. I actually did implement the algorithm to process the entire dataset, but it took my local box about 30 hours just to process the training data, and I did not get to process the test data. This is something I plan to do in the future.
Another strategy I could try is k-fold cross validation, instead of splitting the data into train and validation sets. This way the model can see all of the training data throughout the process, instead of training on the validation data separately. The 3rd place finisher mentioned using 5- fold cross validation in their solution.
I could also try training even more models or different architectures to experiment with larger ensembles. The winner of the competition mentioned having a model for each different label, and the 3rd place finisher used 30 different models and after 5-fold cross-validation resulted in 150 weights files.
Overall, I am extremely happy with my results — achieving top 2% on the leaderboard with an ensemble of only five models. I hope to achieve top 10, or even best the 1st place finisher, by implementing some of the possible areas of improvements listed above.
You can download all of the code to replicate my results from my GitHub.