Mapping Indonesia’s Village Border with Plotly: Transforming Long Lat Coordinates to Shapefile Format
By: Taufiq Bashori
Tags: Indonesia, Geopandas, Geospatial, Python, Plotly

Mapping Indonesia’s Village Border with Plotly: Transforming Long Lat Coordinates to Shapefile Format

DKI Jakarta Map via Plotly (Image by Author)

Plotting geospatial information for Non US locations poses many challenges, ranging from data availability to the heavy lift in data preprocessing. For instance, shapely format for the vector is not always guaranteed, and even so the granularity might not be sufficient.

For a country as vast as Indonesia, having the ability to do spatial analysis at scale can have tremendous benefits for both scholarly and professional work. Therefore, when I came across this GitHub repo with village level GeoJSON granularity (taken from Duscapil), my first response is to fork it.

My aim is simple: to try and make it so that it’s ready for plotting and spatial analysis, for anyone to use. Finding reliable Indonesia’s spatial data reference to be used for analysis has been a challenge for me, and I’m sure some of you might’ve shared similar struggle.

Having said that, this writing mainly focuses on Geopandas and Plotly library, and the methodology is heavily inspired by this TDS post by Kerry Halupka.

Step 1: Data Extraction, Loading & Transformation in Google Colab

First thing First: Retrieve Data Frame from Github Repo

This step should be fairly simple. For the raw dataset, inside the Github repo the owner has given the fairly straightforward parsing method (see here). If you want to recreate the process I’d suggest to do it your local Notebook due to its file size. Extracting the data this way is also convenient as this allows us to locally convert it into csv, which can be then uploaded in database such as BigQuery.

The preview for the raw Data Frame is as follows:

Image by Author (from Github Repo)

Once we have the Data Frame, it’s time to transform it into WKT string geometry. Even though we can do the process locally, Google Colab is hands down better for portability (and more convenient for distribution), so to set it up I uploaded the then-converted-into-csv Data Frame file to google drive and load it into colab environment. Here’s some of the process needed.

Import Necessary Library in Google Colab

The library that we’ll be using are list down below. Import them first to properly setup the colab environment.

Load the Data Frame from Google Drive

As I’ve put the Data Frame in Google Drive, the next step is to mount the source file to drive and specify the path. In this case, my csv file for the Data Frame is located in extract-geojson folder within border-desa-indonesia-geojson folder (within Github May 21 folder inside MyDrive). After we specify the path, we can use Panda’s read_csv to read the file and name it df.

Then I copied the df Data Frame into dfx specifying for the columns that I need. With dfx.head(), if you get the following results, congratulations! You’ve successfully loaded your Data Frame in google colab.

Image by Author

Note: other way to do this is to store your file in github and connect your drive to your github folder, but this will not be covered in this article.

Transform Border Long Lat to Shapely WKT

We’re going to add a column named “geometry” to the data frame, and uses information on the border column to create a polygon or multipolygon which can be converted into wkt string as geometry.

The first step is to create a dictionary to then store the updated polygon/ multipolygon values from the transformed border column. You can choose any column from the dataset but in this case, I’m using border column for the initial dictionary. The use of enumerate ensures that we’re having similar values to the index of the Data Frame.

The second step is to iterate through the border coordinates list values and transform it in a way that’s similar to WKT string Polygon or Multipolygon format. Note that Multipolygon are usually a set of Polygons and can be identified by triple bracket “[[[“ at the beginning of border coordinates. You might want to explore the data to spot more of the subtler differences.

The following syntax has been possible after several iterations to spot both Polygon & Multipolygon while also replaces the irregular characters (such as extra commas or brackets) within each of the “border” column.

The third step is to create the column ‘geometry’ by mapping the data frame index against the newly updated border_dict dictionary. Don’t forget to check on the Multipolygon to confirm if the for loop if logic is working as intended.

If the result is as follows, then congratulations! You’ve loaded the WKT string and is one step closer to having your shapely spatial data reference across (nearly) all of Indonesia.

Image by Author

Error Handlings: How to Know Your Geometry is Valid

So you’ve generated the geometry values and would like to store the data somewhere, maybe in your BigQuery dataset. But how do you know that ALL of the geometry is actually valid?

In order to prevent the hassles of needing to return to data pre-processing, we can iterate through the geometry and check if any error is raised.

If the above code does not generate any error message then you can be sure that the geometry works!

Note: you can also change the None into any values or strings to make your search for error values (if there’s any) be more convenient for you.

Optional: Download Your Updated Data Frame

This optional step is for you who wish to download the data frame. Since we’re using colab, this can be done by files download.

Step 2: Visualize the Map!

Select the Intended Area

First, let’s select in which area of Indonesia do we want to highlight more. For example, say we want to highlight DKI Jakarta, the capital of Indonesia. You can see the code example below.

Here I’m also generating a random integer just to illustrate the use cases (they can be used to color code). In real case, the random integers can be replaced with values/metrics that interest you, i.e populations, households, income, etc. Also don’t forget to also cast the geometry column to WKT geoseries beforehand so that the column data type is recognized as geometry.

Here’s how it translates to after adding pop as column:

Image by Author

Note: if you’d like to visualize all villages in Indonesia it should be no problem, but due to its size you might not be able to do it on google colab.

Optional: Visualize it in GeoPandas

Visualizing it in GeoPandas is much simpler than in Plotly. This optional step is to check quickly the map you’re trying to visualize. Using pop random number as the color code you can run through the following codes:

If you’re running the code and the result is as follows, then you’ve successfully created your GeoPandas map!

DKI Jakarta Map via Geopandas (Image by Author)

Note: Notice there’s a slight missing white vector in the north-middle part of DKI Jakarta. This might indicate that the original dataset does not actually contain geo information on that area. Fixing the white area is out of scope of this writing, but given the small size it shouldn’t be much of an issue.

Visualize it in Plotly

Lastly, here’s the step to visualize it in Plotly. Note that you’re going to need access token before you can proceed. Simply sign up (free) for Mapbox account and generate your key there. There’s also a short tutorial here.

Once you have the key, copy paste it to MAPBOX_ACCESSTOKEN and don’t forget to specify your center starting coordinates. Also as Plotly takes input as geoJSON you should create one from your data frame, in this case I named it x_json.

Finally, once you run the code you should get an beautiful map like this.

DKI Jakarta Map via Plotly (Image by Author)

Now you can simply change your location filter and you’re ready to explore Indonesia with your map! For example, here’s one for Semarang.

Semarang via Plotly (Image by Author)

Happy exploring! Please note however, that not all of the area is properly mapped, i.e for Malang & Surabaya I can still find some missing areas and improperly defined long lat coordinates so exercise with caution :)