Package: bcdata
bcdataPython and command line tools for quick access to DataBC geo-data available via WFS/WCS.There is awealth of British Columbia geographic information available as open
data,
but direct file download urls are not available and the syntax to accesss WFS viaogr2ogrand/orcurl/wgetcan be awkward.This Python module and CLI attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools likegeopandas,fionaandrasterio. As noted above, the module is an interface only to geographic dataavialable via WFS/WCS- to work with other BC open data, use thebcdata R package.DisclaimerIt is the user's responsibility to check the licensing for any downloads, data are generally licensed asOGL-BCDependencies and installationbcdatarequiresgdal. If working with an OS such as linux or MacOS, withgdalalready installed to a known location, install withpip:$ pip install bcdataAlternatively,condacan be used to install/manage the required dependencies as per theGeoPandas guide:$ conda create --name bcdataenv
$ conda activate bcdataenv
$ conda config --env --add channels conda-forge
$ conda config --env --set channel_priority strict
$ conda install python=3 geopandas
$ conda install rasterio
$ pip install bcdataConfigurationDefault PostgreSQL databaseThe default target database connection (used bybc2pg) can be set via theDATABASE_URLenvironment variable (the password parameter should not be required if using a.pgpass file)Linux/Mac:export DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}Windows:SET DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}Layer list / layer schema cacheTo reduce the volume of requests, information about data requested is cached locally.
Schemas of individual layers that have previously been requested are cached with the cache file name matching the object/table name.
Location of the cache defaults to~/.bcdatabut can be modified by setting theBCDATA_CACHEenvironment variable:export BCDATA_CACHE=/path/to/bcdata_cacheThe cache will automatically refresh after 30 days - force a cache refresh by deleting the files in the cache or the entire cache folder.UsageTypical usage will involve a manual search of theDataBC Catalogueto find a layer of interest. Once a dataset of interest is found, note the key with which to retreive it. This can be either theid/package name(the last portion of the url) or theObject Name(UnderObject Description).For example, forBC Airports, either of these keys will work:id/package name:bc-airportsobject name:WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVWNote that some packagesmay have more than one layer- if you request a package like this,bcdatawill prompt you with a list of valid object/table names to use instead of the package name.Python moduleimportbcdata# get a feature as geojsongeojson=bcdata.get_data('bc-airports',query="AIRPORT_NAME='Terrace (Northwest Regional) Airport'")geojson{'type':'FeatureCollection','features':[{'type':'Feature','id':'WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW.fid-f0cdbe4_16811fe142b_-6f34','geometry':{'type':'Point',...# optionally, load data as a geopandas GeoDataFramegdf=bcdata.get_data('bc-airports',query="AIRPORT_NAME='Terrace (Northwest Regional) Airport'",as_gdf=True)gdf.head()AERODROME_STATUSAIRCRAFT_ACCESS_INDAIRPORT_NAME...TC_LID_CODEWEBSITE_URLgeometry0CertifiedYTerrace(NorthwestRegional)Airport...NoneNonePOINT(-128.578333354.46861111)CLICommands available via the bcdata command line interface are documented with the--helpoption:$ bcdata --help

Usage: bcdata [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  bc2pg  Download a DataBC WFS layer to postgres
  cat    Write DataBC features to stdout as GeoJSON feature objects.
  dem    Dump BC DEM to TIFF
  dump   Write DataBC features to stdout as GeoJSON feature collection.
  info   Print basic metadata about a DataBC WFS layer as JSON.
  list   List DataBC layers available via WFSbc2pg$ bcdata bc2pg --help

Usage: bcdata bc2pg [OPTIONS] DATASET

  Download a DataBC WFS layer to postgres

   $ bcdata bc2pg bc-airports --db_url postgresql://postgres:postgres@localhost:5432/postgis

Options:
  -db, --db_url TEXT      Target database url, defaults to $DATABASE_URL
                          environment variable if set
  --table TEXT            Destination table name
  --schema TEXT           Destination schema name
  --geometry_type TEXT    Spatial type of geometry column
  --query TEXT            A valid CQL or ECQL query
  -c, --count INTEGER     Total number of features to load
  -k, --primary_key TEXT  Primary key of dataset
  -s, --sortby TEXT       Name of sort field
  -e, --schema_only       Create empty table from catalogue schema
  -a, --append            Append to existing table
  -r, --refresh           Truncate from existing table before load
  -t, --no_timestamp      Do not log download to bcdata.log
  -v, --verbose           Increase verbosity.
  -q, --quiet             Decrease verbosity.
  --help                  Show this message and exit.cat$ bcdata cat --help

Usage: bcdata cat [OPTIONS] DATASET

  Write DataBC features to stdout as GeoJSON feature objects.

Options:
  --query TEXT                    A valid CQL or ECQL query
  --bounds TEXT                   Bounds: "left bottom right top" or "[left,
                                  bottom, right, top]". Coordinates are BC
                                  Albers (default) or --bounds_crs
  --indent INTEGER                Indentation level for JSON output
  --compact / --not-compact       Use compact separators (',', ':').
  --dst-crs, --dst_crs TEXT       Destination CRS
  -s, --sortby TEXT               Name of sort field
  --bounds-crs, --bounds_crs TEXT
                                  CRS of provided bounds
  -l, --lowercase                 Write column/properties names as lowercase
  -v, --verbose                   Increase verbosity.
  -q, --quiet                     Decrease verbosity.
  --help                          Show this message and exit.dem$ bcdata dem --help

Usage: bcdata dem [OPTIONS]

  Dump BC DEM to TIFF

Options:
  -o, --out_file TEXT             Output file
  --bounds TEXT                   Bounds: "left bottom right top" or "[left,
                                  bottom, right, top]". Coordinates are BC
                                  Albers (default) or --bounds_crs  [required]
  --dst-crs, --dst_crs TEXT       Destination CRS
  --bounds-crs, --bounds_crs TEXT
                                  CRS of provided bounds
  -r, --resolution INTEGER
  -a, --align                     Align provided bounds to provincial standard
  -i, --interpolation [nearest|bilinear|bicubic]
  -v, --verbose                   Increase verbosity.
  -q, --quiet                     Decrease verbosity.
  --help                          Show this message and exit.dump$ bcdata dump --help

Usage: bcdata dump [OPTIONS] DATASET

  Write DataBC features to stdout as GeoJSON feature collection.

    $ bcdata dump bc-airports
    $ bcdata dump bc-airports --query "AIRPORT_NAME='Victoria Harbour (Shoal Point) Heliport'"
    $ bcdata dump bc-airports --bounds xmin ymin xmax ymax

   It can also be combined to read bounds of a feature dataset using Fiona:
   $ bcdata dump bc-airports --bounds $(fio info aoi.shp --bounds)

Options:
  --query TEXT                    A valid CQL or ECQL query
  -o, --out_file TEXT             Output file
  --bounds TEXT                   Bounds: "left bottom right top" or "[left,
                                  bottom, right, top]". Coordinates are BC
                                  Albers (default) or --bounds_crs
  --bounds-crs, --bounds_crs TEXT
                                  CRS of provided bounds
  -l, --lowercase                 Write column/properties names as lowercase
  -v, --verbose                   Increase verbosity.
  -q, --quiet                     Decrease verbosity.
  --help                          Show this message and exit.info$ bcdata info --help

Usage: bcdata info [OPTIONS] DATASET

  Print basic metadata about a DataBC WFS layer as JSON.

  Optionally print a single metadata item as a string.

Options:
  --indent INTEGER  Indentation level for JSON output
  --count           Print the count of features.
  --name            Print the table name of the dateset.
  --description     Print the description of the dateset.
  -v, --verbose     Increase verbosity.
  -q, --quiet       Decrease verbosity.
  --help            Show this message and exit.list$ bcdata list --help

Usage: bcdata list [OPTIONS]

  List DataBC layers available via WFS

Options:
  --help         Show this message and exit.CLI notesNote thatbc2pgcreatesbcdata.logand logs the most recent download date for each table downloaded.
Disable with the switch--no_timestampif you do not wish to create this table.Example of a record inbcdata.log:mydb=# select * from bcdata.log;
                 table_name                  |        latest_download
---------------------------------------------+-------------------------------
 whse_imagery_and_base_maps.gsr_airports_svw | 2021-02-17 11:50:34.044481-08CLI examplesSearch the data listing for airports:$ bcdata list | grep AIRPORTS
  WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVWDescribe a dataset. Note that if we know the id of a dataset, we can use that rather than the object name:$ bcdata info bc-airports --indent 2
{
  "name": "WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW",
  "count": 455,
  "schema": {
    "properties": {
      "CUSTODIAN_ORG_DESCRIPTION": "string",
      "BUSINESS_CATEGORY_CLASS": "string",
      "BUSINESS_CATEGORY_DESCRIPTION": "string",
      "OCCUPANT_TYPE_DESCRIPTION": "string",
      ...etc...
    },
    "geometry": "GeometryCollection",
    "geometry_column": "SHAPE"
  }
}The JSON output can be manipulated withjq. For example, to show only the fields available in the dataset:$ bcdata info bc-airports | jq '.schema.properties'
{
  "CUSTODIAN_ORG_DESCRIPTION": "string",
  "BUSINESS_CATEGORY_CLASS": "string",
  "BUSINESS_CATEGORY_DESCRIPTION": "string",
  "OCCUPANT_TYPE_DESCRIPTION": "string",
  etc...
}Dump data to geojson (EPSG:4326only):$ bcdata dump bc-airports > bc-airports.geojsonGet a single feature and send it to geojsonio (requiresgeojson-cli).  Note the double quotes  required around a CQL FILTER provided to the--queryoption.$ bcdata dump \
  WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW \
  --query "AIRPORT_NAME='Terrace (Northwest Regional) Airport'" \
   | geojsonioSave a layer to a geopackage in BC Albers:$ bcdata cat bc-airports --dst-crs EPSG:3005 \
  | fio collect \
  | fio load -f GPKG --dst-crs EPSG:3005 airports.gpkgNote that this will not work if the source data has mixed geometry types.Load data to postgres and run a spatial query:$ bcdata bc2pg bc-airports \
    --db_url postgresql://postgres:postgres@localhost:5432/postgis

$ bcdata bc2pg WHSE_LEGAL_ADMIN_BOUNDARIES.ABMS_MUNICIPALITIES_SP \
    --db_url postgresql://postgres:postgres@localhost:5432/postgis

$ psql -c \
  "SELECT airport_name
   FROM whse_imagery_and_base_maps.gsr_airports_svw a
   INNER JOIN whse_legal_admin_boundaries.abms_municipalities_sp m
   ON ST_Intersects(a.geom, m.geom)
   WHERE admin_area_name LIKE '%Victoria%'"
                           airport_name
------------------------------------------------------------------
 Victoria Harbour (Camel Point) Heliport
 Victoria Inner Harbour Airport (Victoria Harbour Water Airport)
 Victoria Harbour (Shoal Point) Heliport
(3 rows)Projections / CRSCLIbcdata dumpreturns GeoJSON in WGS84 (EPSG:4326).bcdata catprovides the--dst-crsoption, use any CRS the WFS server supports.bcdata bc2pgloads data to PostgreSQL in BC Albers (EPSG:3005).Python modulebcdata.get_data()defaults toEPSG:4236but any CRS can be specified (that the server will accept).Development and testingbc2pgtests require databasepostgresql://postgres@localhost:5432/test_bcdataexists and has PostGIS installed:psql -c "create database test_bcdata"
psql test_bcdata -c "create extension postgis"Create virtualenv and installbcdatain development mode:$ mkdir bcdata_env
$ virtualenv bcdata_env
$ source bcdata_env/bin/activate
(bcdata_env)$ git clone git@github.com:smnorris/bcdata.git
(bcdata_env)$ cd bcdata
(bcdata_env)$ pip install -e .[test]
(bcdata_env)$ py.testOther implementationsbcdata R package- official BC Open Data R clientOWSLib- used internally bybcdatapgsql-ogr-fdw- read WFS data directly into with postgres foreign data wrappergeneral purpose command line tools(ogr2ogr/curl/wget)