XView 2 Challenge: Part 3: Exploring the Dataset
By: Lezwon Castelino
Tags: Exploratory Data Analysis, Data Science, Computer Vision, Competition, Satellite Imagery

I believe that to train a machine to do something accurately requires an in-depth understanding of the manual task. This outlook has always helped me achieve a better perception of the problem and the nature of the solution. In this series, before we delve into training our model, I’d like to spend some time solving this problem, from the perspective of someone who would have to manually detect and label the right classes for the right pixels. My goal in this post is to get a gist of the dataset and the quality of the images that it comprises.

This post is a follow up to the XView challenge series that I have been writing. If you have not read the previous posts, do consider reading them here as it would provide you some context on this challenge.

I would also like to point out that the submission deadline for the challenge has been closed as of this date of publication. I am however going to continue my attempt on this challenge as it is a very intriguing one. My aim here is to learn the issues faced by experts and to help contribute to solving such real-world challenges in any way possible.

So let’s begin. To help you get started I have created a Kaggle notebook with the code here. Please refer to this notebook for functions and scripts that we are going to be talking about.

As the dataset contains images pertaining to various types of disasters from multiple locations, it would be beneficial to us if we focused on one particular disaster event, as the images from other regions with different disaster types, would give us a radically different building architectures and environments. We shall focus on the dataset pertaining to a particular disaster that occurred in 2011: The Joplin Tornado. This event occurred in 2011 in Missouri, USA and cost damage losses of around $2.8 billion. You can read more about it on Wikipedia.

Loading the data

The first thing we are going to do is load the dataset labels from folders. Notice that the data has been released in two batches: Tier 1 and Tier 3. The data formats for these two releases are essentially the same except for some micro differences in the folder structure.

We are now going to group the labels by the disaster. The `get_diaster_dict` function does the same. Once we receive the dictionary object, we could fetch all the labels associated with a certain disaster. Here we shall fetch all labels related to the Joplin Tornado disaster.

Plotting the images

The next thing I want to do is to compare the pre and post-disaster images. To do this I created a couple of functions that help me to filter and select the right image. The `display_img` function takes in a label path argument and displays the image pertaining to that label. It also has two optional arguments:

display_img(label, time='post', annotated='True') label: String

A path to a json label file. time: ['pre', 'post']

Display pre disaster or post disaster image, depending on the argument. annotated: [True, False]

Displays annotations on the image if argument is set to True

Example:

display_img(disaster_labels[0], time=’post’, annotated=True)

An annotated pre-disaster image of Joplin tornado affected location

An annotated post-disaster image of Joplin tornado affected location

If you notice, in the pre-disaster imagery, the green rooftops are visible and appear to be in a proper condition. In the post-disaster imagery, we notice how most of the rooftops are missing. You can also notice some buildings missing altogether. Our task is to detect such instances of properties, that have been either partially or fully damaged. Let's have a look at more instances and get a better understanding of the differences between the images. The `plot_image` helps us compare such images in a plot. It takes in a label argument and plots all four variations of the image. i.e. the post and pre-disaster image with and without annotations. Here is an example.

Using these functions we can quickly glance through various images in the dataset.

Working with the Metadata

The next thing I wanted was to have a look at the metadata of the images. The metadata for an image is provided within the JSON label files. To get a description of the fields in the metadata, please refer to this document here. To make analyzing this data easier, I wrote a function that converts the data from JSON into a pandas data frame. The `generate_metadata_df` takes in a list of labels and generates a pandas data frame with the fields as columns. I have also added a couple of extra columns here:

centroid_x: Combines the polygons in an image and fetch the central coords (lat) centroid_y: Combines the polygons in an image and fetch the central coords (long) latlong: A WKT format point object with the latlong info. label_path: relative label path ( used for display_img func) no-damage: Count of no-damage polygons in the image minor-damage: Count of minor-damage polygons in the image major-damage: Count of major-damage polygons in the image destroyed: Count of destroyed polygons in the image un-classified: Count of un-classified polygons in the image

Using the centroid data, we can also generate a google maps link to find the location on google maps. The function of `generate_gmaps_link` does the same.

# provides google maps link

generate_gmaps_link(row.centroid_y, row.centroid_x)

Location in Google Maps

Once we have the metadata dataframe we can sort it by any column and iterate to see the examples. For example, I wanted to see images with a lot of destroyed buildings. So I have sorted the dataframe by the destroyed column, and have iterated through the rows to look at the images. Here is an example:

sorted_df = df.sort_values(by=['destroyed'], ascending=False)

index, row = next(df_generator)

plot_image(row.path)

A disaster-affected area with a high number of destroyed polygons

Plotting the Data

When a disaster occurs, there are chances that a particular area might be more affected by it compared to others. Such things vary based on the type of disasters. Floods might cause more damage to coastal areas, tornados will affect the buildings in its path, volcanic eruptions will affect properties surrounding it the most. As such, it would be helpful to map the damages based on the geography of the land. As each polygon has also been mapped using latitude and longitude coordinates, we could use this data to highlight the locations most affected by the disaster.

Referring to a kernel by Alexis Cook I have mapped the disaster affected locations to a map by their damage type. As shown in the map below, you can see a lot of green circles. These circles represent the polygons that are labeled as no-damage. The more the number of polygons of the label type in the image, the greater the size of the circle. You can also notice red circles within the map. These represent the polygons that are labeled as destroyed. The other labels are represented similarly using black, blue and yellow.

A map showing the polygon mapping at Joplin, MO

Please note the circles are drawn at the position of the centroids we calculated earlier for each image. This makes all the polygons in a given image have a common central point which is why they are within each other.

GSD metrics for the Dataset

The dataset has imagery related to a total of 19 different disasters. Each set of images for these disasters has been captured with different cameras having different sensors. Due to this factor, the GSD (Ground Sampling Distance) of the imagery varies to a certain extent. If we are to train this imagery in a neural network, standardizing the GSD across the dataset might help us achieve better accuracy results. The following bar plot shows the mean GSD per image.

Mean GSD values per disaster

As you can see the Guatemala Volcano imagery has the lowest GSD, which means the imagery is closer to the ground and more detailed. If we could somehow utilize super-resolution techniques to get the other dataset imagery to this GSD level, we could train the neural net at a standard GSD level.

Imagery volume per disaster

When we start to train our net, we might have to focus on a particular disaster first. For this, we need to ensure that the selected disaster dataset has abundant training data available. The plot below shows the number of images present per disaster.

Image Count values per disaster

As you can see the Portugal wildfire and the Pinery bushfire have the largest number of images available. The Guatemala volcano, on the other hand, consists of a mere 36 images. We need to take this drawback into account when training the model.

Based on findings in this exercise, we shall proceed to do some image preprocessing on our dataset. In the next post, we will especially try to focus on standardizing GSD across the dataset and also look into some data augmentation options for satellite imagery. If you have any feedback or any ideas for more exploratory data analysis, please do let me know in the comments section.

Please find the code for this post on the Kaggle notebook here: xView 2 Challenge Notebook