Package: geocode-ss
GeocodeGeocode various geographical entities including postcodes and LLSOAs. Reverse-geocode to LLSOA or GSP/GNode.Latest Version: 0.12.1What is this repository for?Use Code Point Open and/or Google Maps API to geocode postcode and/or address into lat/lon co-ordinates.Use ONS & NRS LLSOA Population Weighted Centroids to geocode Lower Layer Super Output Areas.Use GIS data from data.gov.uk to geocode GB constituencies based on geospatial centroid.Use GIS boundaries data from ONS and NRS to reverse-geocode lat/lon to LLSOA.Use GIS data from National Grid ESO's data portal to reverse-geocode to GSP / GNode.Use GIS boudnaries from the Europa/Eurostats API to reverse-geocode to NUTS regions.BenefitsPrioritises Code Point Open for postcode lookup to save expensive GMaps API bills.Caches GMaps API queries locally so that repeated queries can be fulfilled without a new API request.Fetches data automatically at runtime from public APIs where possible.How do I get set up?Developed and tested with Python 3.9, should work for 3.7+.Make sure you have Git installed -Download GitRunpip install geocode-ssorpip install git+https://github.com/SheffieldSolar/Geocode/Check that the installation was successful by running the following command from terminal / command-line:>> geocode -hThis will print the helper for the limited command line interface which provides tools to help get set up and to clear the cache when needed:usage: geocode.py [-h] [--clear-cache] [--debug] [--setup SETUP [SETUP ...]]
                  [--load-cpo-zip </path/to/zip-file>] [--load-gmaps-key <gmaps-api-key>]

This is a command line interface (CLI) for the Geocode module version 0.12.1.

optional arguments:
  -h, --help            show this help message and exit
  --clear-cache         Specify to delete the cache files.
  --debug               Geocode some sample postcodes/addresses/LLSOAs.
  --setup SETUP [SETUP ...]
                        Force download all datasets to local cache (useful if running
                        inside a Docker container i.e. run this as part of image build).
                        Possible values are 'ngeso', 'cpo', 'ons', 'eurostat' or 'all'.
  --load-cpo-zip </path/to/zip-file>
                        Load the Code Point Open data from a local zip file.
  --load-gmaps-key <gmaps-api-key>
                        Load a Google Maps API key.

Jamie Taylor & Ethan Jones, 2019-10-08No additional set up is needed at this stage - the required datasets will be downloaded (or extracted from the packaged data) the first time you use the associated method. If you want to force the Geocode library to download/extract all data, you can run the following command:>> geocode --setupThis is especially useful if you are installing / running the library inside a container - using the above command you can download the data once during the image build rather than have to re-download every time the container is destroyed.ImportantNote that this library makes use of theShapely libraryfrom PyPi, which often does not install correctly on Windows machines due to some missing dependencies. If using Windows and you see an error likeOSError: [WinError 126] The specified module could not be found, you should install Shapely from one of the unofficial binariesheree.g.>> pip install https://download.lfd.uci.edu/pythonlibs/s2jqpv5t/Shapely-1.7.0-cp37-cp37m-win_amd64.whlAll data required by this library is either packaged with the code or is downloaded at runtime from public APIs. Some data is subect to licenses and/or you may wish to manually update certain datasets (e.g. OS Code Point Open) - seeappendix.UsageWithin a Python scriptWithin your Python code, I recommend using the context manager so that GMaps cache will be automatically flushed on exit. Seeexample.py:importosimportloggingfromgeocodeimportGeocoderdefmain():withGeocoder()asgeocoder:# Geocode some postcodes / addresses...print("GEOCODE POSTCODES / ADDRESSES:")postcodes=["S3 7RH","S3 7","S3",None,None,"S3 7RH"]addresses=[None,None,None,"Hicks Building, Sheffield","Hicks","Hicks Building"]results=geocoder.geocode(postcodes,"postcode",address=addresses)forpostcode,address,(lat,lon,status)inzip(postcodes,addresses,results):print(f"    Postcode + Address: `{postcode}` + `{address}`  ->{lat:.3f},{lon:.3f}"f"({geocoder.status_codes[status]})")# Geocode some LLSOAs...print("GEOCODE LLSOAs:")llsoas=["E01033264","E01033262"]results=geocoder.geocode_llsoa(llsoas)forllsoa,(lat,lon)inzip(llsoas,results):print(f"    LLSOA: `{llsoa}`  ->{lat:.3f},{lon:.3f}")# Geocode some Constituencies...print("GEOCODE CONSTITUENCIES:")constituencies=["Sheffield Central","Sheffield Hallam"]results=geocoder.geocode_constituency(constituencies)forconstituency,(lat,lon)inzip(constituencies,results):print(f"    Constituency: `{constituency}`  ->{lat:.3f},{lon:.3f}")# Reverse-geocode some lat/lons to LLSOAs...print("REVERSE-GEOCODE TO LLSOA:")latlons=[(53.384,-1.467),(53.388,-1.470)]results=geocoder.reverse_geocode_llsoa(latlons)forllsoa,(lat,lon)inzip(results,latlons):print(f"    LATLON:{lat:.3f},{lon:.3f}->  `{llsoa}`")# Reverse-geocode some lat/lons to GSP...print("REVERSE-GEOCODE TO GSP:")latlons=[(53.384,-1.467),(53.388,-1.470)]results=geocoder.reverse_geocode_gsp(latlons)for(lat,lon),region_idinzip(latlons,results):print(f"    LATLON:{lat:.3f},{lon:.3f}->{region_id}")# Reverse-geocode some lat/lons to 2021 NUTS2...print("REVERSE-GEOCODE TO NUTS2:")latlons=[(51.3259,-1.9613),(47.9995,0.2335),(50.8356,8.7343)]results=geocoder.reverse_geocode_nuts(latlons,year=2021,level=2)for(lat,lon),nuts2inzip(latlons,results):print(f"    LATLON:{lat:.3f},{lon:.3f}->{nuts2}")if__name__=="__main__":log_fmt="%(asctime)s[%(levelname)s] [%(filename)s:%(funcName)s] -%(message)s"fmt=os.environ.get("GEOCODE_LOGGING_FMT",log_fmt)datefmt=os.environ.get("GEOCODE_LOGGING_DATEFMT","%Y-%m-%dT%H:%M:%SZ")logging.basicConfig(format=fmt,datefmt=datefmt,level=os.environ.get("LOGLEVEL","WARNING"))main()>> python example.py
GEOCODE POSTCODES / ADDRESSES:
    Postcode + Address: `S3 7RH` + `None`  ->  53.381, -1.486 (Full match with GMaps)
    Postcode + Address: `S3 7` + `None`  ->  nan, nan (Failed)
    Postcode + Address: `S3` + `None`  ->  nan, nan (Failed)
    Postcode + Address: `None` + `Hicks Building, Sheffield`  ->  53.381, -1.486 (Full match with GMaps)
    Postcode + Address: `None` + `Hicks`  ->  nan, nan (Failed)
    Postcode + Address: `S3 7RH` + `Hicks Building`  ->  53.381, -1.486 (Full match with GMaps)
GEOCODE LLSOAs:
    LLSOA: `E01033264`  ->  53.384, -1.467
    LLSOA: `E01033262`  ->  53.388, -1.470
GEOCODE CONSTITUENCIES:
    Constituency: `Sheffield Central`  ->  53.376, -1.464
    Constituency: `Sheffield Hallam`  ->  53.396, -1.604
REVERSE-GEOCODE TO LLSOA:
    LATLON: 53.384, -1.467  ->  E01033264
    LATLON: 53.388, -1.470  ->  E01033262
REVERSE-GEOCODE TO GSP:
    LATLON: 53.384, -1.467  ->  ('PITS_3', '_M')
    LATLON: 53.388, -1.470  ->  ('NEEP_3', '_M')
REVERSE-GEOCODE TO NUTS2:
    LATLON: 51.326, -1.961  ->  UKK1
    LATLON: 47.999, 0.234  ->  FRG0
    LATLON: 50.836, 8.734  ->  DE72In the above example,postcodesandaddressesare lists of strings, but it should be fine to use any iterator such as Numpy arrays or Pandas DataFrame columns, although thegeocode()method will still return a list of tuples.When reverse-geocoding to GSP, thereverse_geocode_gsp()method returns both a list of Region IDs and a corresponding list of GSP / GNodes etc. Since the relationship between Region:GSP:GNode is theoretically MANY:MANY:MANY, the second object returned is a list of lists of dicts. This is rather clunky and will likely be refined in a future release. An alternative use case could disregard this second return object and instead make use of theGeocoder.gsp_lookupinstance attribute - this is a Pandas DataFrame giving the full lookup between Regions / GSPs / GNodes / DNO License Areas (i.e.thisdataset on the ESO Data Portal). In testing, thereverse_geocode_gsp()method was able to allocate ~1 million random lat/lons to the correct GSP in average wall-clock time of around 300 seconds.Command Line Utilitieslatlons2llsoaThis utility can be used to load a CSV file containing latitudes and longitudes and to reverse-geocode them to LLSOAs (optionally switching to Data Zones in Scotland):>> latlons2llsoa -h
usage: latlons2llsoa [-h] -f </path/to/file> -o </path/to/file> [--datazones]

This is a command line interface (CLI) for the latlons2llsoa.py module

optional arguments:
  -h, --help            show this help message and exit
  -f </path/to/file>, --input-file </path/to/file>
                        Specify a CSV file containing a list of latitudes and
                        longitudes to be reverse-geocoded. The file must
                        contain the columns 'latitude' and 'longitude' (it can
                        contain others, all of which will be kept).
  -o </path/to/file>, --output-file </path/to/file>
                        Specify an output file.
  --datazones           Specify to use Data Zones in Scotland.

Jamie Taylor, 2020-04-16latlons2gspThis utility can be used to load a CSV file containing latitudes and longitudes and to reverse-geocode them to GSPs/GNodes.>> latlons2gsp -h
usage: latlons2gsp.py [-h] -f </path/to/file> -o </path/to/file>

This is a command line interface (CLI) for the latlons2gsp.py module

optional arguments:
  -h, --help            show this help message and exit
  -f </path/to/file>, --input-file </path/to/file>
                        Specify a CSV file containing a list of latitudes and
                        longitudes to be reverse-geocoded. The file must
                        contain the columns 'latitude' and 'longitude' (it can
                        contain others, all of which will be kept).
  -o </path/to/file>, --output-file </path/to/file>
                        Specify an output file.

Jamie Taylor, 2020-05-29postcodes2latlonThis utility can be used to load a CSV file containing postcodes and to geocode them to lat/lons.>> postcodes2latlon -h
usage: postcodes2latlon.py [-h] -f </path/to/file> -o </path/to/file>

This is a command line interface (CLI) for the postcodes2latlon.py module

optional arguments:
  -h, --help            show this help message and exit
  -f </path/to/file>, --input-file </path/to/file>
                        Specify a CSV file containing a list of postcodes to
                        be geocoded. The file must contain the column
                        'postcode' (it can contain others, all of which will
                        be kept).
  -o </path/to/file>, --output-file </path/to/file>
                        Specify an output file.

Jamie Taylor, 2020-06-12bng2latlonThis utility can be used to load a CSV file containing eastings and northings and to convert them to lat/lons.>> bng2latlon -h
usage: bng2latlon.py [-h] -f </path/to/file> -o </path/to/file>

This is a command line interface (CLI) for the bng2latlon.py module

optional arguments:
  -h, --help            show this help message and exit
  -f </path/to/file>, --input-file </path/to/file>
                        Specify a CSV file containing a list of eastings and
                        northings to be converted. The file must contain the
                        columns 'eastings' and 'northings' (it can contain
                        others, all of which will be kept).
  -o </path/to/file>, --output-file </path/to/file>
                        Specify an output file.

Jamie Taylor, 2020-06-12DocumentationSeehere.How do I update?Runpip install --upgrade git+https://github.com/SheffieldSolar/Geocode/.You may also wish to clear all locally cached data by running the following command from terminal/command-prompt:geocode --clear-cacheDevelopersRunning Tests>> cd <root-dir>
>> python -m unittest Tests.test_geocodeAppendixCode Point OpenAs of version 0.4.15, the Code Point Open data is packaged with this repository, however the following instructions can be used to update the local Code Point Open dataset without re-installing the Geocode library.Contains OS data © Crown copyright and database right 2018Contains Royal Mail data © Royal Mail copyright and Database right 2018Contains National Statistics data © Crown copyright and database right 2018Updating Code Point OpenDownload the (free) Code Point Open datahere.
Run the following command:>> geocode --load-cpo-zip "<path-to-zip-file>"For example, if you saved the code point open data to 'C:\Users\jamie\Downloads\codepo_gb.zip', you would run:>> geocode --load-cpo-zip "C:\Users\jamie\Downloads\codepo_gb.zip"N.B. If you encounter problems using the--load-cpo-ziputility, you might like to manually copy/paste the zip filecodepo_gb.zipinto thegeocode/code_point_opensubdirectory of the geocode module. (Do not extract the contents of the zipfile)The CPO data will be extracted and stored locally ingeocode/code_point_open/code_point_open_<version>.p.GMapsIf you want to make use of the Google Maps API, you can load an API key using the command:>> geocode --load-gmaps-key "<your-api-key>"N.B. The API key will be stored in plaintext in thegeocode/google_maps/key.txtfile within your local installation. This is not secure and only appropriate when using the Geocode library for development purposes.Seeherefor help getting an API key.Any queries you make to the GMaps API will be cached togeocode/google_maps/gmaps_cache_<version>.pso that repeated queries are faster and cheaper.ONSThe Geocode library makes use of data from the Office for National Statistics in order to geocode Lower Layer Super Output Areas (LLSOAs) in England and Wales. The first time you make use of theGeocoder.geocode_llsoa()method, the LLSOA (December 2011) Population Weighted Centroids data will be downloaded from the ONS API and cached locally ingeocode/ons_nrs/llsoa_centroids_<version>.p. The first time you make use of theGeocoder.reverse_geocode_llsoa()method, the LLSOA (December 2011) Boundaries EW data will be downloaded from the ONS API and cached locally ingeocode/ons_nrs/llsoa_boundaries_<version>.p. More informationhereandhere.NRSThe Geocode library makes use of data from National Records of Scotland in order to geocode Lower Layer Super Output Areas (LLSOAs) in Scotland. The raw population-weighted centroids and boundary data is available from the NRS website, but for convenience and performance reasons the data has been re-formatted and re-projected and is packaged with this library. More informationhere.Contains NRS data © Crown copyright and database right 2020Contains Ordnance Survey data © Crown copyright and database right 2020data.gov.ukThe Geocode library makes use of the Westminster Parliamentary Constituencies (December 2015) Generalised Clipped Boundaries in Great Britain† from data.gov.uk in order to geocode constituencies according to their (unweighted) geospatial centroid. More informationhere. The geospatial centroids have been calculated using ESRI ArcMap's 'Feature To Point' toolbox and the derived lookup data is packaged within this respository as a pipe-separated variable file:geocode/gov/constituency_centroids.psv. The first time you make use of theGeocoder.geocode_constituency()method, this file will be loaded and cached back togeocode/gov/constituency_centroids_<version>.pfor faster loading next time.†Contains public sector information licensed under the Open Government Licence v3.0.NGESO Data PortalThe Geocode library makes use of GSP/GNode GIS boundaries developed by Sheffield Solar. In May 2020, these region definitions were uploaded to National Grid ESO's Data Portal - seehere. The first time you make use of theGeocoder.reverse_geocode_gsp()method, the GIS data is downloaded from the Data Portal API at runtime.Supported by National Grid ESO Open DataSubject toNGESO Open LicenceTo update your locally cached boundary definitions, clear your local cache:geocode --clear-cacheEurostatThe Geocode library makes use of the eurostat API to download boundaries for NUTS (Nomenclature of territorial units for statistics) regions. For more information, seehere. For license conditions, see the FAQ section of the Europa websitehere.