Deep Learning with Python and ArcGIS: Detecting Solar Panels
By: Ablajan Sulaiman
Tags: Machine Learning, GIS, Deep Learning, Artificial Intelligence, Python

Introduction

AI has become more popular today due to ever-increasing data volumes, advanced algorithms, and improvements in computing power and storage. The integration of artificial intelligence (AI) with GIS (geographic information systems) creates a new scientific discipline so-called geospatial artificial intelligence (GeoAI). By integrating machine learning with GIS technologies, it is possible to perform insightful spatial big data analyses to deal with unprecedented traffic, pollution, energy demands, environmental issues, public health, and safety concerns in modern cities.

I was recently involved with a solar mapping project about assessing the solar energy potential of building roofs and providing information about solar panel suitability and its associated benefits in Toronto. I found that solar panels have already been installed on some of the building rooftops. Out of curiosity, I tried Deep Learning to find how many buildings equipped with solar panels.

There are a lot of tutorials and papers about detecting photovoltaic solar panels using areal or satellite images. Here, I’ll introduce how to detect solar panels using Deep Learning in ArcPro.

Deep learning in ArcGIS Pro

ArcGIS Pro, Server, and the ArcGIS API for Python all include tools to use AI and Deep Learning to solve geospatial problems, such as feature extraction, pixel classification, and feature categorization. ArcGIS Pro allows us to use statistical or machine learning classification methods to classify remote-sensing imagery. Deep learning models can be integrated with ArcGIS Pro for object detection, object classification, and image classification. ArcGIS deep learning toolset contains tools to detect specific features in an image or to classify pixels in a raster dataset. Here are three steps of deep learning workflow such as preparing training data, training a model, and deploying the trained model. Let’s get started by installing Deep Learning libraries for ArcGIS Pro.

Installing Deep Learning Libraries

All deep learning geoprocessing tools in ArcGIS Pro require that the supported deep learning frameworks libraries be installed. Each version of ArcGIS Pro requires specific versions of deep learning libraries which include a broad collection of python packages, such as PyTorch, TensorFlow, Fast.ai, and scikit-learn, for performing deep learning and machine learning tasks.

Installing Deep Learning libraries for ArcGIS Pro is very straightforward. The Deep Learning Libraries Installer can be downloaded from Esri GitHub (Deep Learning Libraries Installer). For instructions on how to install deep learning packages, see the Deep Learning Libraries Installer for ArcGIS Pro.

Installing Deep Learning Libraries

Or any other specific python libraries can be installed by using ArcPro Python Package Manager in a cloned active python virtual environment or by executing python package installation commands (pip, conda) in the active python virtual environment. Each version of ArcGIS Pro requires specific versions of deep learning libraries.

(deeplearning) C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning>conda install -c esri -c fastai -c pytorch arcgis=1.8.1 scikit-image=0.15.0 pillow=6.2.2 libtiff=4.0.10 fastai=1.0.60 pytorch=1.4.0 torchvision=0.5.0 tensorflow-gpu=2.1.0

Python Package Manager

Once we’ve installed the deep learning libraries, we can use the Deep Learning Tools to train geospatial deep learning models. Let’s get started by preparing training sample data.

Data Preparation: Create classes and label objects

All supervised deep learning tasks depend on labelled datasets. Image annotation, or labelling, is vital for deep learning tasks such as computer vision and learning. It can be done by using Label Objects for Deep Learning and the Export Training Data For Deep Learning tools in ArcPro. The Label Objects for Deep Learning tool is found in the Classification Tools drop-down menu, in the Image Classification group on the Imagery tab. Let’s create a new schema and add solar_panel image class by using the image classification tool as below:

Image Classification

Then we can select the solar_panel image class from a random image and start sketching solar panel shape as shown on the image below.

Creating Training Samples

A large amount of labelled data is required to train a good deep learning model. Multiple solar panels can be sketched on some building rooftops.

Data preparation in Deep Learning, especially processing image data, is a time-consuming process. In order to speed up creating training samples, a few hundred samples are created at first.

Solar_Panel Samples

We can use the Export Training Data For Deep Learning tool to convert the samples into deep learning training data.

Export Training Data

The output is a folder of image chips and a folder of metadata files in the specified format. This tool creates training datasets to support third-party deep learning applications, such as Google TensorFlow, Keras, PyTorch, and Microsoft CNTK.

Training Datasets

Training a Model

Use the Train Deep Learning Model tool to train a model using PyTorch, or train the model outside of ArcGIS Pro using a supported third-party deep learning framework in Jupyter notebook. Let’s start training a model by using the training datasets created in the first step.

# Import necessary ArcGIS Python API tools:

from arcgis.gis import GIS

from arcgis.raster.functions import apply

from arcgis.learn import export_training_data

from arcgis.learn import SingleShotDetector, prepare_data

Let’s define the dataset folder.

data_path = r”C:\esri_deeplearning\deep_learning\deeplearning”

prepare_data function prepares a data object from the training sample exported by the Export Training Data. Let’s check the classes.

data = prepare_data(data_path)

# Checking on the classes loaded with the data:

data.classes

Checking Data Classes

The prepare_data function used approximately 10 percent of the image chips to train the model to recognize solar panels. The other 90 percent is used to test the model’s accuracy. The prepared data can be checked by visualizing a random batch from the training set.

data = prepare_data(data_path, {1:’solar_panel’}, batch_size=32)

arcgis.learn allows us to define SSD architecture just through a single line of code. Let’s define a Single Shot Detector with the specified grid sizes, zoom scales and aspect ratios.

from arcgis.learn import SingleShotDetector

ssd = SingleShotDetector(data, grids=[4], zooms=[1.0], ratios=[[1.0, 1.0]])

The grid parameter specifies the size of the grid cell, in this case, 4x4. arcgis.learn includes learning rate finder, and is accessible through the model's lr_find() method, which can automatically select an optimum learning rate, without requiring repeated experiments. Additionally, we are specifying a zoom level of 1.0 and an aspect ratio of 1.0:1.0. lr_find() function runs the Learning Rate Finder and displays the graph of its output.

SSD Learning Rate

The learning rate is approximately 0.0017. model.fit() function can be called to fit our model to the training data. Here we are training the model for 10 epochs.

ssd.fit

Typically validation loss should be similar to but slightly higher than training loss. It seems the above-trained model is a good fit.

What is the average precision score for the solar panel detections at this point?

ssd.average_precision_score()

It is obvious that the 80% accuracy rate is pretty high at our first attempt with fewer training samples.

ssd.show_results(rows=25, thresh=0.05)

Ground truth\ Predictions

Let’s save the model in the model's folder.

ssd.save(r”C:\dev\solar”)

Trained Model

Now the trained model is ready to be deployed.

Deploying Trained Model

The above-trained model can be used with the Detect Objects Using Deep Learning tool in ArcPro to generate an output. The detect_objects() can be used to generate feature layers that contain a bounding box around the detected objects in the image data using the trained deep learning model.

Let’s test a single image. The parallel processing factor can be used to speed up the geoprocessing.

Detect Solar Panels in Single Image

The python script of the above process can be executed in Jupyter notebook.

#python script for single image object detection

import arcpy

arcpy.CheckOutExtension("ImageAnalyst")

from arcpy.ia import *

# Execute arcpy.ia.ExportTrainingDataForDeepLearning(“test_image”, r”C:\Users\asulaim\Documents\solar\soldata”, r”C:\dev\solar\solar_panel.shp”, “JPEG”, 256, 256, 128, 128, “ONLY_TILES_WITH_FEATURES”, “PASCAL_VOC_rectangles”, 0, “Classvalue”, 0, None, 0, “MAP_SPACE”, “PROCESS_AS_MOSAICKED_IMAGE”, “NO_BLACKEN”, “FIXED_SIZE”)

As shown in the following image, the solar panel detection results (highlighted in orange colour)are pretty accurate.