Geospatial adventures. Step 2: Pandas vs. GeoPandas | by Dmitry Selemir | Towards Data Science

Having introduced shapely in my first post, it’s time to look at some interesting geo datasets and in order to do that we can not possibly do without Pandas and more specifically GeoPandas. I am going to assume you have come across Pandas before and will try and highlight some of the differences between the two and more importantly how you can convert one to the other.
We start from the very beginning, if you have never worked with GeoPandas before — go ahead and install it, especially if you would like to follow the steps from this post yourself and play with the data a little bit. Good old trusted pip is all you need.
!pip install geopandas
If you are so inclined (and you absolutely should be) — you can read the docs here.
Let’s import both Pandas and GeoPandas to start.
import pandas as pdimport geopandas as gpd
Now, before we can do anything with these two, we need a dataset to play with. There are some links to interesting geo-specific datasets at the end of this article (do check them out), but let’s start with something relatively short and yet realistic enough: UK local authority boundaries. The reason I like this dataset is, on the one hand, it has just 380 records, on the other — the boundaries and shapes it is using are anything but simple, so it is a good illustration of a dataset you are likely going to work with in real life. I also think this is a really good starting point for any geospatial analysis of the UK data as it allows you to break everything into manageable chunks. Trust me — it really helps at times.
So, without further ado, the dataset lives here. Just hit the download button on your right. You’ll get a zip file with a short and sweet name Local_Authority_Districts__December_2017__Boundaries_in_Great_Britain-shp.zip weighing at approximately 35mb (at list at the time of writing, they do get updates occasionally ). Go ahead and unzip it and you’ll get a folder looking like this:
What we are after is the shape file, the one with *.shp extension.
To the best of my knowledge, regular Pandas can not deal with that format, so here comes our first difference between Pandas and Geopandas as, of course, the latter can. Let’s go ahead and open it:
la = gpd.read_file('Downloads/Local_Authority_Districts__December_2017__Boundaries_in_Great_Britain-shp/Local_Authority_Districts__December_2017__Boundaries_in_Great_Britain.shp')
Make sure you are using the path relevant to your system. If you are unsure which folder your notebook is using by default just run <!ls> in one of the cells — you’ll get the current directory contents, so you should be able to work out where to go from here. If you need to go up a folder — use <../>.
Let’s take a peek at what the data looks like. All the standard Pandas commands work, so we can just do:
la.head()
To get
Nice! OK, what the hell does it actually mean?
Let’s go through it column by column: objectid is fairly self explanatory and, frankly, not very useful to us. lad17cd is the local authority code. This is actually quite handy as it is used alongside things like super output areas by ONS, so this gives us a good reference system to go back and forth between different geographical subdivisions and corresponding datasets. Non-UK people — I do apologise for being so partisan here, the truth is, most countries will have a somewhat similar way of looking at things, so even if you find this is not directly of interest, this can still be transferable and useful. lad17nm: local authority names. Some of these are weird and wonderful, so it is absolutely worth exploring. No? Just me then...lad17nmw, which seems to be populated by friendly looking <None>s (sorry, couldn’t resist). This column also has names, and “w” kind of gives it away — it is local authority names in Welsh, hence None values in the first five records as these are all English local authorities. We don’t get out of England until number 317, in case you were wondering.bng_e and bng_n — the evil twins. Seriously, these are Eastings/Northings— I’ll talk about these a little bit more later. Basically, coordinates showing where this area is on the map.long and lat — some more coordinates — longitude and latitude this time.st_sreasha — surface area in square meters, that “sha” at the end is somewhat confusing, but it means shape.st_lengths — boundary length (also in meters).And finally, the one we are really here for! Drum roll please....geometry — this is the actual polygon or, in some cases, multiPolygon shape. Don’t get me started on the Scottish isles.... Geometry is the key attribute of GeoPandas table and many applications working with them essentially assume that this column is there and that it is called exactly “geometry”. So if, for whatever reason, you wanted to give it another, friendlier name — think again.
Before we dive in any further, since we are comparing Pandas and GeoPandas among other things, we can create a copy of this table in regular Pandas format.
la_pd = pd.DataFrame(la)la_pd.head()
You’ll notice that this new table looks absolutely identical. Not only that, the objects in geometry table are still very much intact:
So far so good. We can also look at all the stats for the DataFrame in the same way we do it in Pandas and the two tables will look identical, so I’m going to show the results for GeoPandas only here (you will just have to believe me... No? OK, run it in your own notebook for both just to check).
Note, this one gives you the stats on numerical data columns only.
In[9]:
la.info()
Out[9]:
<class 'geopandas.geodataframe.GeoDataFrame'>RangeIndex: 380 entries, 0 to 379Data columns (total 11 columns):objectid      380 non-null int64lad17cd       380 non-null objectlad17nm       380 non-null objectlad17nmw      22 non-null objectbng_e         380 non-null int64bng_n         380 non-null int64long          380 non-null float64lat           380 non-null float64st_areasha    380 non-null float64st_lengths    380 non-null float64geometry      380 non-null geometrydtypes: float64(4), geometry(1), int64(3), object(3)memory usage: 32.7+ KB
And finally:
In[10]:
la.memory_usage()
Out[10]:
Index           80objectid      3040lad17cd       3040lad17nm       3040lad17nmw      3040bng_e         3040bng_n         3040long          3040lat           3040st_areasha    3040st_lengths    3040geometry      3040dtype: int64
So far, so good. Time to spot another difference. This command would not work on your Pandas DataFrame:
In[11]:
la.crs
Out[11]:
<Projected CRS: EPSG:27700>Name: OSGB 1936 / British National GridAxis Info [cartesian]:- E[east]: Easting (metre)- N[north]: Northing (metre)Area of Use:- name: UK - Britain and UKCS 49°46'N to 61°01'N, 7°33'W to 3°33'E- bounds: (-9.2, 49.75, 2.88, 61.14)Coordinate Operation:- name: British National Grid- method: Transverse MercatorDatum: OSGB 1936- Ellipsoid: Airy 1830- Prime Meridian: Greenwich
Woah!!! What does that mean?! Remember I promised to expand on Northings/Eastings? Time to look at the coordinate reference systems, aka CRS. You can read a more technical GeoPandas specific explanation here. In a nutshell, defining coordinate systems is not as straightforward as it may sound at first. The main issue here is that the Earth has the audacity of not being flat. Not only that — it’s not even spherical, so defining an exact point on the surface can be tricky (if you don’t believe me — try wrapping a square-lined paper around a rugby ball). To make matters even more interesting, because it is so large — it sometimes makes sense to pretend it is flat because, on the scale even large enough to cover an entire country like the UK, this doesn’t introduce enough error to really worry about. Hence two main reference systems you are likely going to come across — longitude and latitude (pseudo 3D) and Northings/Eastings (2D). I have to confess that, in most cases, I prefer to work with the latter. The reason being — it’s easy and I am lazy. Seriously though, both Northings and Eastings are expressed in meters and measure a straightforward distance. So, if you want to know how far things are from each other, all you have to do is take the differences between Xs (eastings) and Ys(northings), apply Pythagorus and you are done. Note that Shapely library, which we talked about in my previous post, also has the concept of CRS built in, however, we don’t really need to use it, as GeoPandas provides us with all the ammunition we could possibly need, with high precision.
There is a number of CRS systems available in GeoPandas. This particular one, used in the table above is “epsg:27700” — if you are planning to work with the UK data this is something you are going to use a lot, so you’ll get to remember it, be warned. An alternative in the UK, using lat/long is “epsg:4326”. GeoPandas provides us with an easy way of converting one to the other. All we have to do is run:
la_4326 = la.to_crs("epsg:4326")la_4326.head()
To get:
The eagle-eyed among you would have spotted that the only thing that’s changed in this table is our polygon/multiPolygon object column, with each object now composed of points referencing latitudes and longitudes. You can still look at the same graphical representation as we did before and it should look identical. I will leave that for you to verify.
Before we wrap with some calculations on top of these — let’s have a look at saving to csv and converting from Pandas to GeoPandas. In more recent versions of Pandas you should be able to save the DataFrame directly to a csv file, despite the geometric objects in the last column, i.e.
la.to_csv('Downloads/la.csv', compression='gzip')
will do the trick. What’s more, the resulting file is only 39.9mb vs. 63.7mb for the shapefile (of course, we already applied compression here though). It comes at a price, however. To the best of my knowledge, you can not read a csv file directly from GeoPandas, so you have to load it back as a normal DataFrame. Simply by running:
la_new = pd.read_csv('Downloads/test.csv', compression='gzip')
At first glance, nothing much has changed:
la_new.head()
Essentially, we have a single extra column, which we can easily get rid of by running
la_new = la_new[la_new.columns[1:]]
However, this is not all of it, if we try to have a look at one of our geometry objects as we did before, we get something very different:
In[17]
la_new['geometry'].iloc[0]
Out[17]:
'MULTIPOLYGON (((447213.8995000003 537036.1042999998, 447228.7982999999 537033.3949999996, 447233.6958999997 537035.1045999993, 447243.2024999997 537047.6009999998, 447246.0965 537052.5995000005, 447255.9988000002 537102.1953999996, 447259.0988999996 537108.8035000004, 447263.6007000003 537113.8019999992, 447266.1979 537115.6015000008, 447273.1979999999 537118.6007000003, 447280.7010000004 537120.1001999993, 447289.3005999997 537119.6004000008, 447332.2986000003 537111.5026999991, 447359.5980000002 537102.3953000009, 447378.0998999998 537095.0974000003, 447391.0033999998 537082.9009000007, 447434.6032999996 537034.5046999995, 447438.7011000002 537030.9956999999, 447443.7965000002 537027.6966999993...
OK, I’m cheating here, in reality, the output is a lot longer than that. Essentially what happened is all our geometry objects got converted to strings. All is not lost, however, as shapely provides us with a way of converting them back. Apart from loading shapely here, I will also need swifter library. You can do without it by using straight apply method, but among other things — swifter provides you with a nice progress bar and time estimate (it can also improve performance, but that is beyond the scope of this post).
import swifterfrom shapely import wktla_new['geometry1'] = la_new['geometry'].swifter.apply(lambda x: wkt.loads(x))
What’s happening here is I am applying the conversion function to each element of the geometry column and store the output in the new column. On my laptop this takes 12 seconds on our 380 record long dataset. I have done this on much, much larger datasets and to be fair it’s not too bad.
We can then spot check if the new elements are indeed geometry objects by running
la_new['geometry1'].iloc[0]
We are not done yet though. In fact, if we run la_new.info() we get the following:
<class 'pandas.core.frame.DataFrame'>RangeIndex: 380 entries, 0 to 379Data columns (total 12 columns): #   Column      Non-Null Count  Dtype  ---  ------      --------------  -----   0   objectid    380 non-null    int64   1   lad17cd     380 non-null    object  2   lad17nm     380 non-null    object  3   lad17nmw    22 non-null     object  4   bng_e       380 non-null    int64   5   bng_n       380 non-null    int64   6   long        380 non-null    float64 7   lat         380 non-null    float64 8   st_areasha  380 non-null    float64 9   st_lengths  380 non-null    float64 10  geometry    380 non-null    object  11  geometry1   380 non-null    object dtypes: float64(4), int64(3), object(5)memory usage: 35.8+ KB
The new column appears as an object type, not geometry type. This won’t stop us from converting it to GeoDataFrame however. All we have to do is get rid of extra geometry column we no longer need and rename geometry1 to geometry (remember — it has to be named that) and then (!) and this is important, after converting to GeoDataFrame we have to set crs for it,
la_new_geo = la_new.drop(columns=['geometry']).rename(columns={'geometry1': 'geometry'})la_new_geo = gpd.GeoDataFrame(la_new_geo)la_new_geo.crs = 'epsg:27700'
And voila! la_new_geo.info() gives us:
<class 'geopandas.geodataframe.GeoDataFrame'>RangeIndex: 380 entries, 0 to 379Data columns (total 11 columns): #   Column      Non-Null Count  Dtype   ---  ------      --------------  -----    0   objectid    380 non-null    int64    1   lad17cd     380 non-null    object   2   lad17nm     380 non-null    object   3   lad17nmw    22 non-null     object   4   bng_e       380 non-null    int64    5   bng_n       380 non-null    int64    6   long        380 non-null    float64  7   lat         380 non-null    float64  8   st_areasha  380 non-null    float64  9   st_lengths  380 non-null    float64  10  geometry    380 non-null    geometrydtypes: float64(4), geometry(1), int64(3), object(3)memory usage: 32.8+ KB
Finally, some quick calculations on our new shiny GeoDataFrame:
Verifying length and area:
la_new_geo['area']=la_new_geo['geometry'].swifter.apply(lambda x: x.area)la_new_geo['length'] = la_new_geo['geometry'].swifter.apply(lambda x: x.length)la_new_geo[['lad17cd', 'lad17nm', 'st_areasha', 'st_lengths', 'area', 'length']].head()
Or if we wanted to verify the entire column, we could do:
(la_new_geo.st_lengths == la_new_geo.length).all()
You’ll notice that this would actually return False however, due to rounding error. For instance, on the first record, the st_lengths value is 71707.4075227013 and length value is 71707.40752270141
So to check we can just run it with set precision:
(la_new_geo.st_lengths.round(4)==la_new_geo.length.round(4)).all()
In any case, when dealing with geographical objects, moving beyond 25cm precision is probably unjustified given the quality of existing data and for most applications you are probably OK limiting it to the nearest meter anyway.
Let’s also have a look at which region has the most complicated polygon/multiPolygon. We are going to judge that by the number of points needed to describe it. Essentially, this means counting points in exterior and interior boundaries:
la_new_geo['point_count'] = la_new_geo['geometry'].swifter.apply(    lambda x: np.sum([len(np.array(a.exterior)) + np.sum([len(np.array(b)) for b in a.interiors]) for a in x]) if x.type == 'MultiPolygon'     else len(np.array(x.exterior)) + np.sum([len(np.array(a)) for a in x.interiors]))
Ok, this is a bit complicated, so let’s look at what is going on. First we are looking at MultiPolygon objects and if we have one — we have to iterate through each polygon, converting it to exterior boundary and then numpy array to get an array of coordinates. We take the length of that array. We also get the list of internal boundaries, so we have to also iterate through them, convert them to arrays and take length. Similarly, if we are dealing with a simple polygon we drop one level of iteration, so just one exterior boundary, but we still need to iterate through potentially multiple holes.
We then sort the values and voila, we have our top ten:
la_new_geo[    ['lad17cd', 'lad17nm', 'point_count']].sort_values('point_count', ascending=False).head(10)
And this is why everyone loves Scotland so much.
No? Just me then...
Next post is all about matching polygons with each other, the coolest invention of mankind — R-Trees (Erm.. sort of). Well... OK, maybe one or two pictures too.
And finally the promised...
OSM datasets for all of the World: http://download.geofabrik.de/
OSM datasets for Britain: http://download.geofabrik.de/europe/great-britain.html
ONS Open Data: https://www.ordnancesurvey.co.uk/opendatadownload/products.html
NOMIS (labour market /census data):https://www.nomisweb.co.uk/ — these usually will have data for specific area, which you can then relate to the map by using their boundary shape files (matching them by ID)
I’m sure there’s many, many more, please feel free to add any interesting geospatial datasets (preferably free to use) in the comments.
See you around...
Also in this series:
Geospatial adventures. Step 1: Shapely.
Geospatial adventures. Step 3. Polygons grow on R-Trees
Geospatial adventures. Step 4. The Colour of Magic or If I Don’t See It — It Doesn’t Exist
Geospatial adventures. Step 5. Leaving the flatlands or flying over the sea of polygons