Spatial Data Solution for City Planning in Indonesia: Understanding The GeoDataFrame | by Sutan Ashari Mufti | Towards Data Science

Geographic Information System (GIS) is an essential skill an urban planner must-have. With GIS, urban planners may manage, publish, and analyze spatial data. The spatial data is distinct to ordinary data as spatial data has a spatial element that sometimes people overlook. The awareness and intuition about the “spatial” are what makes an urban planner, or geographer powerful.
As an Indonesian Urban Planner, I deal with spatial data every day, and this essay is inspired by a dataset that is published by Jakarta City, Indonesia. I must say, Jakarta City’s spatial data repository impresses me! It is structured, readable, processable, and well maintained. You do not need digitation. It is a great example of the spatial data repository that many boroughs must look up to. But there’s this one “spatial” data from the Jakarta’s Open Data Portal that raises questions (at least, to me): The Routes of Public Transportation.
The routes are described in a manner that is understandable for people, but not parsable for computers in a geographical sense. For example, where is “Jl. Raya Bekasi” street? How long it is? How does it curve? What are the geometrical properties? To a data analyst like me, this is infuriating. We must look for the street, plot it in GIS software so that we can do further analysis with the computer. This (digitation) process spends time, and time is money. The unprepared data costs analysts and produces tedious works.
Let’s look at another example: a map like illustrated by the figure below. This is supposed to be a map of an Angkot (“public transportation — Minibus”) route.
It is not a map per se, it is a diagram. Geometrical properties in the “map” are just a diagrammatical representation. The length and distance are not the true scaled distance, this is why the “map” does not have a coordinate system/reference.
An attempt to include spatial data such as illustrated above should be appreciated. It holds information, but we can do more.
Now, how should we analyze this spatial data computationally? If you’ve worked with much data especially spatial data, I’m sure you quite understand what I’m talking about. Imagine the digitizing, and cleaning the data. This is not efficient, and only one person/entity should do the cleansing so that not everyone has to. The solution is simple, the provision of standardized, formated, and parsable spatial data.
The spatial data has so many formats. Most popular, I guess, is the Shapefile (.shp) format developed by Esri in the '90s which is pretty old! This is 2020, it’s been a decade since the era of shapefile, surely many data and techniques have been developed. As I play around with Python, I noticed a connection between GeoDataFrame and the spatial data problems that I encountered. So, this essay introduces another type of geospatial data, GeoDataFame, and how it can solve the spatial data attribute problem for Indonesia.
This essay consists of 5 parts:
What is GeoDataFrame? (explains what is GeoDataFrame and how it is different with conventional spreadsheets)Why GeoDataFrame? (the reason for choosing GeoDataFrame, because it is intuitive to most people)Storing The Geometrical Attribute (showing the manifestation/form of the geometrical data. We will use Shapely in Python, but don’t worry about Python, you don’t have to understand Python)How we Store and Distribute the GeoDataFrame (so that people that are familiar and not familiar with spatial data can still understand the data)Concluding Remarks (summarizing and recommendation of what we should do to get started)
What is GeoDataFrame? (explains what is GeoDataFrame and how it is different with conventional spreadsheets)
Why GeoDataFrame? (the reason for choosing GeoDataFrame, because it is intuitive to most people)
Storing The Geometrical Attribute (showing the manifestation/form of the geometrical data. We will use Shapely in Python, but don’t worry about Python, you don’t have to understand Python)
How we Store and Distribute the GeoDataFrame (so that people that are familiar and not familiar with spatial data can still understand the data)
Concluding Remarks (summarizing and recommendation of what we should do to get started)
GeoDataFrame is a spatial data frame that is used by GeoPandas as an object to operate. When we use GeoPandas to analyze and manage spatial data, GeoDataFrame is the target object. The official explanation of GeoDataFrame from GeoPandas is
“A GeoDataFrame is a tabular data structure that contains a GeoSeries.” — GeoPandas
what is a GeoSeries? GeoSeries is a vector that contains the geometries of the data. We can imagine a vector as a single column spreadsheet. It is represented as a column named “geometry” in a GeoDataFrame. I always think of the definition to be confusing and visualizing it always helps. Here’s what a GeoDataFrame look like:
As you can see, a GeoDataFrame is basically a spreadsheet that contains one column of geometry. I’d like to say that GeoDataFrame is like rows of data that have a geometry property. Each feature (each row) has values and a geometric property. For instance, as illustrated, “Pulau Tidung” has a population of 11305 with a geometry described in the geometry column.
When we plot or do spatial analysis, this “geometry” column is the one that is being processed. Dissecting the GeoDataFrame there are 3 components:
The DataFrame: this is the rows and columns of the data. We can imagine this like a Ms. excel spreadsheet. Here we can statistically describe and analyze the features. For instance, what village has the highest population? How’s each village’s population density? (but not spatial interaction)
The Geometry: this is a column that comprises corresponding geometry for each row (feature). This is what defines the spatial element of the data
The Coordinate Reference: Geographical reference of the geometry. For instance, a point with coordinate x=8 and y=15; where is the 0 here that each X and Y can refer? Commonly we use WGS 1984 (World Geodetic System, EPSG 4326) or as most people know, the Longitude and Latitude. You can see that in the first figure, in the “geometry” column, it holds Longitude and Latitude values).
We can also plot the geometry line and make a map. In this example, the population of Jakarta City. Instead of graphs, we can use Choropleth map so that we can overview the spatial relationships.
Geodataframe is more readable to more common people. GeoDataFrame is basically a table, and everyone knows what a table is, so it’s intuitive. Unlike Shapefile that, I think, not everyone knows.
If you’re familiar with python, it’s basically a Pandas Dataframe with one additional column that holds the geometric properties. We can do Pandas methods to GeoDataFrame.
Now, what is the value is inside this geometry column? We’re going to use some Python nomenclatures, but it’s not that complicated as it is pretty much straight forward. Please bear with me a little.
There are 3 main spatial data types:
Points: this is pretty straight forward. Points indicate the location and toponyms. Points are one dimensional and contain 1 coordinate. (expect multipoint, which has many coordinates with the same attributes, but they don’t makeup lines or areas)
Polylines: Lines or Polylines are sets of coordinates that are connected by a line. We can use Polylines to describe networks such as pipes, roads, or rail networks.
Polygon: Polygon is enclosed lines so it has an area. Used to describe administrative boundaries, zoning regulations, basis of a choropleth map.
So Spatial data is basically a bunch of coordinates!
By their nature, Polylines and Polygons consist of vertices. These vertices are Points’ coordinates that are held in the Geometry column. As we can see in the figure below when a value of geometry is expanded, it contains many coordinates of Latitude and Longitude, and when we visualize these vertices, it draws a shape.
Each geometry is an object of Shapely, a library of Python. Shapely deals with geometrics for math purposes. We can do much spatial analysis such as intersections, unions, and many more. This is a very brief introduction: for example, if I want to create a geometric object, what I need to do is write some command to Python like this:
>> from shapely.geometry import Polygon>> Polygon([(104.4, -6.2),(106.4, -7.0), (103.2, -8.9)]
This will create a rectangle as there are 3 vertices (3 coordinates).
Of course, to define each feature’s geometry we do not put it manually. Instead, we can integrate sensors such as GPS to automate the task or do some conversions with some scripts.
While shapefile is very popular I think GeoJSON is neater. Shapefile consists of separate files, besides the geometry itself, such as the projections and the attributes; while GeoJSON is only one “text” file.
Web service that fulfills the OGC standard is a must, this allows websites and algorithms to connect remotely, but sometimes users need downloadable data with common and popular formats such as .xlsx or .csv.
There are several options that I recommend:
Save it as CSV: because the spatial data must be accessible to many people from many backgrounds, the simplest way to store is to convert the GeoDataFrame to a CSV (comma-separated values). This way, spreadsheet software can parse the CSV to become a tablature format. The caveat is that the “geometry” column will be stored as text, so to convert it to the geometric (shapely) object we need further parsing. A web application or some simple script should do the job. It’s an option, but I don’t think this option is preferred.
Save the Dataframe as CSV, and Geometry column as a separate GeoJSON: This yields 2 files; a CSV and a GeoJSON. Don’t worry about GeoJSOn, it’s basically a text file. GeoJSON is a format for interoperability between systems or software. With GeoJSON, many software or websites can parse the text and plot the geometries. The GeoJSON objects will be indexed so that it can be joined with CSV.
I think there should be an awareness of spatial attributes in datasets. Spatial attributes are as crucial as the data frame itself, and should not be separated. A spreadsheet (.xlsx, CSV, many more) must be accompanied by geometry in a GeoJSON format, or even embedded in the CSV itself. A tabular data format must have a column that defines the geometry of each feature whenever geometric properties are suitable. Why GeoDataFrame, CSV or GeoJSON? So it is readable to human eyes to those who are not common to spatial data, and common to spatial data. The geometry must consist of coordinate(s).
I think Including GeoDataFrame should increase the intuition to many people that features do have spatial attributes, and spatially interactable. GeoDataFrame also enables us to compute spatial data in an efficient way. To get started, We can do as simple as “only” adding one column to each of our spreadsheets, that is, the geometry column.