Package: geocode-sqlite
geocode-sqliteGeocode rows from a SQLite tableInstallationInstall this tool usingpiporpipx:# install inside a virtualenvpipinstallgeocode-sqlite# install globallypipxinstallgeocode-sqliteUsageLet's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
First, create a SQLite database and insert rows from that spreadsheet usingsqlite-utils.sqlite-utilsinsertdata.dbdatadata.csv--csvNow, geocode it using OpenStreetMap's Nominatim geocoder.geocode-sqlitenominatimdata.dbdata\--location="{address}, {city}, {state} {zip}"\--delay=1\--user-agent="this-is-me"In the command above, you're using Nominatim, which is free and only asks for a unique user agent (--user-agent).This will connect to a database (data.db) and read all rows from the tabledata(skipping any that already
have both alatitudeandlongitudecolumn filled).You're also telling the geocoder how to extract a location query (--location) from a row of data, using Python's
built-in string formatting, and setting a rate limit (--delay) of one request per second.For each row where geocoding succeeds,latitudeandlongitudewill be populated. If you hit an error, or a rate limit,
run the same query and pick up where you left off.The resulting table layout can be visualized withdatasette-cluster-map.Under the hood, this package uses the excellentgeopylibrary, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consultgeopy's documentation.Supported GeocodersThe CLI currently supports these geocoders:binggooglev3mapquest(andopen-mapquest)mapboxnominatimopencageAdding new geocodersOpen an issue with the name of the geocoding service as the ticket title (example). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.Fork the repo and add a geocoder.Add an example to theMakefile. Add tests if there's new shared functionality.Common arguments and optionsEach geocoder needs to know where to find the data it's working with. These are the first two arguments:database: a path to a SQLite file, which must already existtable: the name of a table, in that database, which exists and has data to geocodeFrom there, we have a set of options passed to every geocoder:location: astring formatthat will be expanded with each row to build a full query, to be geocodeddelay: a delay between each call (some services require this)latitude: latitude column namelongitude: longitude column namegeojson: store results as GeoJSON, instead of in latitude and longitude columnsspatialite: store results in a SpatiaLite geometry column, instead of in latitude and longitude columnsraw: store raw geocoding results in a JSON columnEach geocoder takes additional, specific arguments beyond these, such as API keys. Again,geopy's documentationis an excellent resource.Using SpatiaLiteThe--spatialiteflag will store results in ageometry column, instead oflatitudeandlongitudecolumns. This is useful if you're doing other GIS operations, such as using aspatial index. See theSpatiaLite cookbookandfunctions listfor more of what's possible.Capturing additional geocoding dataGeocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the--rawflag. By default, this will add arawcolumn and store the full geocoding response as JSON. If you want to rename that column, pass a value, like--raw custom_raw.The shape of this response object will vary between services. You can query specific values usingSQLite's built-in JSON functions. For example, this will work with Google's geocoder:selectjson_extract(raw,'$.formatted_address')asaddress,json_extract(raw,'$.geometry.location_type')aslocation_typefrominnout_testCheck each geocoding service's documentation for what's included in the response.Python APIThe command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.As with the CLI, this assumes you already have a SQLite database and a table of location data.fromgeocode_sqliteimportgeocode_tablefromgeopy.geocodersimportNominatim# create a geocoder instance, with some extra optionsnominatim=Nominatim(user_agent="this-is-me",domain="nominatim.local.dev",scheme="http")# assuming our database is in the same directorycount=geocode_table("data.db","data",query_template="{address},{city},{state}{zip}")# when it's doneprint(f"Geocoded{count}rows")Anygeopy geocodercan be used with the Python API.DevelopmentTo contribute to this tool, first checkout the code. Then create a new virtual environment:cdgeocode-sqlite
python-mvenv.venvsource.venv/bin/activateOr if you are usingpipenv:pipenvshellNow install the dependencies and tests:pipinstall-e'.[test]'To run the tests:pytestPlease remember that this library is mainly glue code between other well-tested projects, specifically:click,geopyandsqlite-utils. Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.To that end, there is a test geocoder included:geocode_sqlite.testing.DummyGeocoder. That geocoder works with an included dataset of In-N-Out Burger locations provided byAllThePlaces. It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.