Package: intake-dcat
intake-dcatThis is anintakedata source forDCATcatalogs.These catalogs are a standardized format for describing metadata and access information
for public datasets, as describedhere.
Many Socrata and ESRI data portals publishdata.jsonfiles in this format describing their catalogs.
Two examples of thes can be found athttps://data.lacity.org/data.jsonhttp://geohub.lacity.org/data.jsonThis project provides an opinionated way for users to load datasets from these catalogs into the scientific Python ecosystem.
At the moment it loads CSVs into Pandas dataframes and GeoJSON files into GeoDataFrames, and ESRI Shapefiles into GeoDataFrames.
Future formats could include plain JSON and Parquet.Requirementsintake >= 0.4.4
intake_geopandas >= 0.2.2
geopandas >= 0.5.0Installationintake-dcatis published on PyPI.
You can install it by running the following in your terminal:pipinstallintake-dcatYou can test the functionality by opening the example notebooks in theexamples/directoryUsageThe package can be imported usingfromintake_dcatimportDCATCatalogLoading a catalogYou can load data from a DCAT catalog by providing the URL to thedata.jsonfile:catalog=DCATCatalog('http://geohub.lacity.org/data.json',name='geohub')len(list(catalog))You can display the items in the catalogforentry_id,entryincatalog.items():display(entry)If the catalog has too many entries to comfortably print all at once,
you can narrow it by searching for a term (e.g. 'district'):forentry_id,entryincatalog.search('district').items():display(entry)Loading a datasetOnce you have identified a dataset, you can load it into a dataframe usingread():df=entry.read()This will automatically load that dataset into a Pandas dataframe, or a GeoDataFrame, depending on the source format.Specifying catalogsYou can read aDCATCatalogdirectly in Python using a URL, as done above,
but it is also possible to write a catalog file that itself containsDCATCatalogentries.
This allows you to more easily specify DCAT catalogs for use in distribution and version control.For instance, this YAML file creates entries for two open data catalogs:metadata:version:1sources:# Here we have two data sources for this catalog, which are themselves# DCAT catalogs, one for LA open data, and the other for LA GeoHubla_open_data:# We identify them as being loaded with the DCAT driverdriver:dcat# Here we specify the args used to load the catalogargs:# The URL to the catalogurl:https://data.lacity.org/data.json# An optional name for the catalog.name:la-open-datala_geohub:driver:dcatargs:url:http://geohub.lacity.org/data.jsonname:la_geohub# We can also specify a subset of the datasets in the catalog using an "items"# dictionary. If these are specified, only these datasets will be available in# the resulting catalog. They will be available under the more human-readable# name specified as the key.items:# So, this dataset will be available as "bikeways"bikeways:http://geohub.lacity.org/datasets/2602345a7a8549518e8e3c873368c1d9_0city_boundary:http://geohub.lacity.org/datasets/09f503229d37414a8e67a7b6ceb9ec43_7bike_racks:http://geohub.lacity.org/datasets/3b022cced9704108af157d3d5eedb268_2Command Line Interfaceintake-dcatprovides a small command line interface for some common operations.
These are invoked usingintake-dcat <subcommand> <options>ThemirrorcommandThis command loads a manifest file that lists a set of DCAT entries,
uploads them to a specified s3 bucket, and outputs a new catalog with identical entries
pointing to the bucket.An example manifest is given by# Name of the LA open data portal
la-open-data:
  # URL to the open data portal catalog
  url: https://data.lacity.org/data.json
  # The s3 bucket to upload the data to
  bucket_uri: s3://my-bucket
  # A list of data resources to mirror
  items:
    lapd_metrics: https://data.lacity.org/api/views/t6kt-2yic
# Name of the LA GeoHub data portal
la-geohub:
  # URL to the open data portal catalog
  url: http://geohub.lacity.org/data.json
  # The s3 bucket to upload the data to
  bucket_uri: s3://my-bucket
  # A list of data resources to mirror
  items:
    bikeways: http://geohub.lacity.org/datasets/2602345a7a8549518e8e3c873368c1d9_0 
    city_boundary: http://geohub.lacity.org/datasets/09f503229d37414a8e67a7b6ceb9ec43_7This can be mirrored using the commandintake-dcatmirrormanifest.yml>new-catalog.ymlThis command uses theboto3library and assumes it can find AWS credentials.
For more information seethis documentation.ThecreatecommandThis command creates a new intake catalog from a DCAT catalog, and outputs it to standard out.
An example command is given byintake-dcatcreatedata.lacity.org/data.json>catalog.yml