GIS data processing cheat sheet: Effectively using command line GDAL | by Kanishka Narayan | Towards Data Science

Geographical Information systems (GIS) are highly informative and provide a granular representation of data. Needless to say, these datasets are very attractive to researchers, data analysts and data scientists. But these data come with a big catch. They are heavy and a pain to manipulate from a processing and computational time perspective. Given their size and structure, it is also difficult to analyze these datasets meaningfully unless you are using software like ArcGis or QGIS. In this article, I provide some simple command line code that can be used to greatly minimize the processing time and can be effectively used to extract meaningful information from these data. The code makes use of the GDAL program that is available here. So, let’s jump right in.
To those unfamiliar with GIS data types- I’d like to provide a simple summary of the different types of GIS data that one may encounter. There are two broad categories to consider. The first is raster data which are grid/cell based data that contain some information about what a cell contains (discrete data like the population contained within the cell) or what it represents (continuous data like what category a particular cell belongs to). An example of a raster format would be a .tif, .bsq or .bil file. The second is vector data which are data represented as shapes like maps of countries, states etc. An example of a vector format is a .shp or shapefile. Now that, we know the data we will be working with, let’s jump into the code.
To use this code, you will need GDAL installed on your machine. If you are on a windows machine, I would recommend installing GDAL through a cygwin terminal.
1. Getting information/metadata- I have used an example raster for this article here, called gpw_v4_population_count_rev11_2000_30_sec.tif. This is a file downloaded from a database maintained by NASA that can be accessed here.It is a simple raster that contains population by grid cell for every country in the world. To see what this raster actually contains let’s use the below gdalinfo function.
#First go the directory containing the filecd 'file path here'#Now let's run gdalinfogdalinfo gpw_v4_population_count_rev11_2000_30_sec.tif#This is the output we getDriver: GTiff/GeoTIFFFiles: gpw_v4_population_count_rev11_2015_30_sec.tifSize is 43200, 21600Origin = (-180.000000000000000,89.999999999999915)Pixel Size = (0.008333333333333,-0.008333333333333)Metadata:AREA_OR_POINT=AreaDataType=GenericImage Structure Metadata:COMPRESSION=LZWINTERLEAVE=BANDCorner Coordinates:Upper Left  (-180.0000000,  90.0000000)Lower Left  (-180.0000000, -90.0000000)Upper Right ( 180.0000000,  90.0000000)Lower Right ( 180.0000000, -90.0000000)Center      (  -0.0000000,  -0.0000000)Band 1 Block=43200x1 Type=Float32, ColorInterp=GrayNoData Value=-3.40282306073709653e+38
So, the information we get tells us the Driver: which is the type of file (which in this case is a tif file), Size: which is the size of the grid (the number of rows and columns which make up the grid which in this case are 43200,21600). The Pixel Size: which is the size of the projection within a pixel (which in this case is 0.00833333 degrees which is 5 arc minutes), the Corner Co-ordinates: which is the extent of the projection (This will tell us if certain regions are missing from the projection. For example, some maps will have some regions cut off etc) and finally we get the NoData value: which is the value with no data accounting, or essentially the nulls within the datasets. All of these data are useful as we will see in the rest of the article. But gdalinfo is really useful in that it provides a quick summary of the available data within a file and consumes no processing power.
2. Filling Nodata values in a raster- For different projects, we may not want NoData values or may want to replace NoData values with a different value from the default. There are 3 steps that we can to follow to accomplish this with gdal. First, we need to replace the NoData value with our own default (let’s say 0), replace all NoData values with zeroes (This is a sanity check to ensure that we are catching Nodata values that are not specified by the metadata) and then we need to take off the NoData tag, since NoData is essentially a qualitative tag. We can use two commands here, first the gdalwarp to replace values and a gdal_translate to remove the tag. Note that both these commands like most other gdal commands will ask you to specify a separate output object.
#Fill NoData values with zerogdalwarp -srcnodata -3.40282306073709653e+38 dstnodata 0 gpw_v4_population_count_rev11_2015_30_sec.tif population_nodatafilled.tif# Now replace all no_data values with a 0.gdal_translate -b 1 -a_nodata 0 population_nodatafilled.tif population_only_data.tif#Note that the -b indicates the band within the raster where you want to the operation. This would change for multiband rasters.# Now take off the no_data tag itselfgdal_translate -a_nodata none population_only_data.tif population_clean.tif
3. Re-projecting rasters- When comparing two raster files, we need to make sure we are comparing apples to apples since there may be differences in the grid size, projection type or projection sizes or even the extents. The gdalwarp allows us to do convenient reprojections to standardize raster files. So, continuing on with our current example, if we wanted to reproject our data onto a smaller grid size with a different projection type, we would use the following gdalwarp command. The comment in the code also provides a description of the parameters we are feeding this command.
#Let's reproject gdalwarp -ts 1440 600 -r average -ot Float32 -t_srs "+proj=longlat +ellps=WGS84" -te -180 -60 180 90 population_clean.tif population_reprojected.tif#Note the following:# -ts is the size of the grid that we want in rows and columns# -r is specifying the method used to aggregate/dis-aggregate. There #are a number of options available, but the one that makes the most #sense is average since this is basically the sum of the cell values #being aggregated divided by the number of cells.# -ot is the output type which is set to a Float since this us a #mathematical operation.# -t_srs is the reprojection type. The one I am specifying here is a #longlat projection using a WGS84 datum.# -te is the extent itself, which is the extent of the projection. #If you standardize the grid size and the reprojection type, you will automatically standradize the pixel size.   
Now, you may have noticed that you can’t even open the original tif. After the reprojection though, we can actually plot out the raster since the size is much smaller.
4. Convert raster data types- Often users will want to convert data types. A common reason to do this would be space. For example, geotiffs are expensive in terms of memory but .bil or .bsq files are much cheaper. So, we can convert this easily using gdal_translate. The great part about this is that gdal will create header files with automatically where ever necessary.
#Convert tif to bilgdal_translate -of ENVI population_reprojected.tif population_final.bil#The -of specifies the file type we want to convert to.
5. Replacing specific pixel values based on a criteria- Now, users may also want to replace specific values within a raster with their own values. Let’s say we wanted in the raster in question, grid cells with any value of population to have a value of 1 while those without should have a value of 0. We can achieve this using gdal_calc.py. We would just use the below code to complete this. Note that this is a python command.
#Replace pixel values based on criteriagdal_calc.py -A population_final.bil --calc='((A>0))*1+((A<=0)*A)' --outfile=population_final_reclass.bil#This will basically replace all values within pixels with 1 and those without values to 0.
6. Raster to vector conversion (polygonization)- Let’ say we wanted a global map of regions according to population. That is, a map that showed us where people were actually living in the world. We already have a raster which tracks exactly this for each pixel.If we could combine the pixels to get a meaningful map, that would be very helpful. This is easy to achieve in gdal using gdal_polygonize.py. This would take a raster, convert pixels to polygons and merge neighbouring pixels with the same values (in this case 1) and output a global map.
#Convert raster to vector (shape file)gdal_polygonize.py population_final_reclass.bil population_map.shp pop#Note that the pop is the destination column name within the shape file. 
After this, if we were to plot out the map of the world regions where people live we get the map below. The great part about this is that it will correspond to the dimensions of our raster.
7. Merging vector files- Now, let’s say in the above map, we also wanted a representation in the map above of the different countries in the world. With the ogr2ogr command, you can also merge two vector files. Let’s say we had a shapefile called countries.shp with the actual boundaries of countries. We could merge it with our existing shape file to get a map file that shows you a map of countries along with holes for places within countries where people don’t live.
#First create a file to merge againstogr2ogr -f 'ESRI Shapefile' population_boundaries_merged.shp population_map.shp#Now merge the boundaries shapefile into this oneogr2ogr -f -update -append population_boundaries_merged.shp ccountries.shp -nln merge#This command is updating the population_boundaries_merged.shp by appending a new boundaries file from countries.shp. 
And, this is the output we get out of the new population_boundaries_merged
There you have it. These commands are a useful cheatsheet when having to process heavy GIS data and extract meaningful information out of them. They can obviously be modified to be adapted to a number of operations. I am also in the process of developing a repository that will host these commands as a formal set of bash scripts. The repository in its current form is here. As always, any and all feedback related to this is welcome!