Package: amsterdam-schema-tools
amsterdam-schema-toolsSet of libraries and tools to work with Amsterdam schema.Install the package with:pip install amsterdam-schema-tools. This installs
the library and a command-line tool calledschema, with various subcommands.
A listing can be obtained fromschema --help.Subcommands that talk to a PostgreSQL database expect either aDATABASE_URLenvironment variable or a command line option--db-urlwith a DSN.Many subcommands want to know where to find schema files. Most will look in a
directory of schemas denoted by theSCHEMA_URLenvironment variable or the--schema-urlcommand line option. E.g.,schema create tables --schema-url=myschemas mydatasetwill try to load the schema formydatasetfrommyschemas/mydataset/dataset.json.Generate amsterdam schema from existing database tablesThe --prefix argument controls whether table prefixes are removed in the
schema, because that is required for Django models.As example we can generate a BAG schema. PointDATABASE_URLtobag_v11database and then run :schema show tablenames | sort | awk '/^bag_/{print}' | xargs schema introspect db bag --prefix bag_ | jqThejqformats it nicely and it can be redirected to the correct directory
in the schemas repository directly.Express amsterdam schema information in relational tablesAmsterdam schema is expressed as jsonschema. However, to make it easier for people with a
more relational mind- or toolset it is possible to express amsterdam schema as a set of
relational tables. These tables aremeta_dataset,meta_tableandmeta_field.It is possible to convert a jsonschema into the relational table structure and vice-versa.This command converts a dataset from an existing dataset in jsonschema format:schema import schema <id of dataset>To convert from relational tables back to jsonschema:schema show schema <id of dataset>Generating amsterdam schema from existing GeoJSON filesThe following command can be used to inspect and import the GeoJSON files:schema introspect geojson <dataset-id> *.geojson > schema.json
edit schema.json  # fine-tune the table names
schema import geojson schema.json <table1> file1.geojson
schema import geojson schema.json <table2> file2.geojsonImporting GOB eventsThe schematools library has a module that reads GOB events into database tables that are
defines by an Amsterdam schema. This module can be used to read GOB events from a Kafka stream.
It is also possible to read GOB events from a batch file with line-separeted events using:schema import events <path-to-dataset> <path-to-file-with-events>Export datasetsDatasets can be exported to different file formats. Currently supported are geopackage,
csv and jsonlines. The command for exporting the dataset tables is:schema export [geopackage|csv|jsonlines] <id of dataset>The command has several command-line options that can be used. Documentations about these
flags can be shown using the--helpoptions.Schema Tools as a pre-commit hookIncluded in the project is apre-commithook
that can validate schema files
in a project such asamsterdam-schemaTo configure it
extend the.pre-commit-config.yamlin the project with the schema file defintions as follows:-repo:https://github.com/Amsterdam/schema-toolsrev:v3.5.0hooks:-id:validate-schemaargs:['https://schemas.data.amsterdam.nl/schema@v1.2.0#']exclude:|(?x)^(schema.+|             # exclude meta schemasdatasets/index.json)$argsis a one element list
containing the URL to the Amsterdam Meta Schema.validate-schemawill only processjsonfiles.
However not alljsonfiles are Amsterdam schema files.
To exclude files or directories useexcludewith pattern.pre-commitdepends on properly tagged revisions of its hooks.
Hence, we should not only bump version numbers on updates to this package,
but also commit a tag with the version number; see below.Doing a release(This is for schema-tools developers.)We use GitHub pull requests. If your PR should produce a new release of
schema-tools, make sure one of the commit increments the version number insetup.cfgappropriately. Then,merge the commit in GitHub, after review;pull the code from GitHub and merge it into the master branch,git checkout master && git fetch origin && git merge --ff-only origin/master;tag the release X.Y.Z withgit tag -a vX.Y.Z -m "Bump to vX.Y.Z";push the tag to GitHub withgit push origin --tags;release to PyPI:make upload(requires the PyPI secret).Mocking dataThe schematools library contains two Django management commands to generate
mock data. The first one iscreate_mock_datawhich generates mock data for all
the datasets that are found at the configured schema locationSCHEMA_URL(whereSCHEMA_URLcan be configure to point to a path at the local filesystem).Thecreate_mock_datacommand processes all datasets. However, it is possible
to limit this by adding positional arguments. These positional arguments can be
dataset ids or paths to the location of thedataset.jsonon the local filesystem.Furthermore, the command has some options, e.g. to change
the default number of generated records (--size) or to reverse meaning of the positional
arguments using--exclude.To avoid duplicate primary keys on subsequent runs the--start-atoptions can be used
to start autonumbering of primary keys at an offset.E.g. to generate 5 records for thebagandgebiedendatasets, starting the
autonumbering of primary keys at 50.django create_mock_data bag gebieden --size 5 --start-at 50To generate records for all datasets, except for thefietspaaltjesdataset:django create_mock_data fietspaaltjes --exclude  # or -xTo generate records for thebbgadataset, by loading the schema from the local filesystem:django create_mock_data <path-to-bbga-schema>/datasets.jsonDuring record generation increate_mock_data, the relations are not added,
so foreign key fields will be filled with NULL values.There is a second management commandrelate_mock_datathat can be used to
add the relations. This command support positional arguments for datasets
in the same way ascreate_mock_data.Furthermore, the command also has the--excludeoption to reverse the meaning
of the positional dataset arguments.E.g. to add relations to all datasets:django relate_mock_dataTo add relations forbagandgebiedenonly:django relate_mock_data bag gebiedenTo add relations for all datasets exceptmeetbouten:django relate_mock_data meetbouten --exclude  # or -xNB. When only a subset of the datasets is being mocked, the command can fail when datasets that
are involved in a relation are missing, so make sure to include all relevant
datasets.For convenience an additional management commandtruncate_tableshas been added,
to truncate all tables.