Package: pdok-geopackage-validator
PDOK geopackage-validatorGeopackages are a data format that have a deliberately broad application, so many of the requirements are dependend on your use.The PDOK geopackage validator is used byPDOK. PDOK is part of the Dutch government. This geopackage validator is used to validate aset of requirementsto make sure geopackages adhere to our standardized ETL pipeline. It is possible to use this for your own purposes as describedhere. The validations will not change (except for bugfixes);new validations are always added to the list.Table of Contentsgeopackage-validatorTable of ContentsWhat does it doGeopackage versionsInstallationDockerUsageRQ8 ValidationShow validationsGenerate table definitionsLocal developmentUsagePython consoleCode styleTestsReleasingTL;DR CommandsEither run throughdockerorlocally.DockerValidate a GeoPackage with the default set of validation rules:gpkg_path=relative/path/to/the.gpkg
dockerrun-v"$(pwd)":/gpkg--rmpdok/geopackage-validatorvalidate--gpkg-path"/gpkg/${gpkg_path}"Validate a GeoPackage with the default set of validation rules including a schema:schema_path=relative/path/to/the/schema.jsongpkg_path=relative/path/to/the.gpkg
dockerrun-v"$(pwd)":/gpkg--rmpdok/geopackage-validatorvalidate--gpkg-path"/gpkg/${gpkg_path}"--table-definitions-path"/gpkg/${schema_path}"Generate a schema:schema_path=relative/path/to/the/schema.jsongpkg_path=relative/path/to/the.gpkg
dockerrun-v"$(pwd)":/gpkg--rmpdok/geopackage-validatorgenerate-definitions--gpkg-path"/gpkg/${gpkg_path}">"$schema_path"LocalFor a local setup we require/tested against python > 3.6 and gdal = 3.4.gpkg_path=relative/path/to/the.gpkg
geopackage-validatorvalidate--gpkg-path"/gpkg/${gpkg_path}"Validate a GeoPackage with the default set of validation rules including a schema:schema_path=relative/path/to/the/schema.jsongpkg_path=relative/path/to/the.gpkg
geopackage-validatorvalidate--gpkg-path"/gpkg/${gpkg_path}"--table-definitions-path"/gpkg/${schema_path}"Generate a schema:schema_path=relative/path/to/the/schema.jsongpkg_path=relative/path/to/the.gpkg
geopackage-validatorgenerate-definitions--gpkg-path"/gpkg/${gpkg_path}">"$schema_path"What does it doThe Geopackage validator can validate .gkpg files to see if they conform to a set of standards.
The current checks are (see also the 'show-validations' command):Validation code**DescriptionUNKNOWN_ERRORNo unexpected (GDAL) errors must occur.RQ0LEGACY:* Geopackage must conform to table names in the given JSON or YAML definitions.RQ1Layer names must start with a letter, and valid characters are lowercase a-z, numbers or underscores.RQ2Layers must have at least one feature.RQ3LEGACY:* Layer features should have an allowed geometry_type (one of POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, or MULTIPOLYGON).RQ4The geopackage should have no views defined.RQ5Geometry should be valid.RQ6Column names must start with a letter, and valid characters are lowercase a-z, numbers or underscores.RQ7Tables should have a feature id column with unique index.RQ8Geopackage must conform to given JSON or YAML definitions.RQ9All geometry tables must have an rtree index.RQ10All geometry table rtree indexes must be valid.RQ11OGR indexed feature counts must be up to date.RQ12Only the following EPSG spatial reference systems are allowed: 28992, 3034, 3035, 3038, 3039, 3040, 3041, 3042, 3043, 3044, 3045, 3046, 3047, 3048, 3049, 3050, 3051, 4258, 4936, 4937, 5730, 7409.RQ13It is required to give all GEOMETRY features the same default spatial reference system.RQ14The geometry_type_name from the gpkg_geometry_columns table must be one of POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, or MULTIPOLYGON.RQ15All table geometries must match the geometry_type_name from the gpkg_geometry_columns table.RQ16All layer and column names shall not be longer than 53 characters.RC17It is recommended to name all GEOMETRY type columns 'geom'.RC18It is recommended to give all GEOMETRY type columns the same name.RC19It is recommended to only use multidimensional geometry coordinates (elevation and measurement) when necessary.RC20It is recommended that all (MULTI)POLYGON geometries have a counter-clockwise orientation for their exterior ring, and a clockwise direction for all interior rings.UNKNOWN_WARNINGSIt is recommended that the unexpected (GDAL) warnings are looked into.* Legacy requirements are only executed with the validate command when explicitly requested in the validation set.** Since version 0.8.0 the recommendations are part of the same sequence as the requirements. From now on a check will always maintain the integer part of the code. Even if at a later time the validation type can shift between requirement and recommendation.An explanation in Dutch with a reason for each rule can be foundhere.Geopackage versionsThe Geopackage validator support the following Geopackage versions:1.3.11.31.2.1InstallationThis package requires:GDALversion >= 3.2.1.Spatialiteversion >= 5.0.0And python >= 3.8 to run.We recommend using the docker image. When above requirements are met the package can be installed using pip (pip install pdok-geopackage-validator).Docker InstallationPull the latest version of the Docker image (only once needed, or after an update)dockerpullpdok/geopackage-validator:latestOr build the Docker image from source:dockerbuild-tpdok/geopackage-validator.The command is directly called so subcommands can be run in the container directly:dockerrun-v${PWD}:/gpkg--rmpdok/geopackage-validatorvalidate-t/path/to/generated_definitions.json--gpkg-path/gpkg/tests/data/test_allcorrect.gpkgUsageRQ8 ValidationTo validate RQ8 you have to generate definitions first.dockerrun-v${PWD}:/gpkg--rmpdok/geopackage-validatorgeopackage-validatorgenerate-definitions--gpkg-path/path/to/file.gpkgValidateUsage:geopackage-validatorvalidate[OPTIONS]GeopackagevalidatorvalidatingalocalfileorafilefromS3storage.Whenthefilepathisprecededwith'/vsis3'or'/vsicurl'thegdalvirtualfilesystemwillbeusedtoaccessthefileonS3andwillnotbedirectlydownloaded.Seehttps://gdal.org/user/virtual_file_systems.htmlforfurtherexplanationhowtousegdalvirtualfilesystems.ForconveniencethegdalvsienvironmentparametersandoptionalparametersareprovidedwithanS3_insteadofanAWS_prefix.TheAWS_environmentparameterswillalsowork.Examples:viscurl:geopackage-validatorvalidate--gpkg-path/vsicurl/http://minio-url.nl/bucketname/key/to/public.gpkgvsis3:geopackage-validatorvalidate--gpkg-path/vsis3/bucketname/key/to/public.gpkg--s3-signing-regioneu-central-1--s3-secret-keysecret--s3-access-keyacces-key--s3-secure=false--s3-virtual-hostingfalse--s3-endpoint-no-protocolminio-url.nlS3_SECRET_KEY=secretS3_ACCESS_KEY=acces-keyS3_SIGNING_REGION=eu-central-1S3_SECURE=falseS3_VIRTUAL_HOSTING=falseS3_ENDPOINT_NO_PROTOCOL=minio-url.nlgeopackage-validatorvalidate--gpkg-path/vsis3/bucketname/key/to/public.gpkgAWS_SECRET_ACCESS_KEY=secretAWS_ACCESS_KEY_ID=acces-keyAWS_DEFAULT_REGION=eu-central-1AWS_HTTPS=NOAWS_VIRTUAL_HOSTING=FALSEAWS_S3_ENDPOINT=minio-url.nlgeopackage-validatorvalidate--gpkg-path/vsis3/bucketname/key/to/public.gpkg

Options:--gpkg-pathFILEPathpointingtothegeopackage.gpkgfile[envvar:GPKG_PATH]-t,--table-definitions-pathFILEPathpointingtothetable-definitionsJSONorYAMLfile(generatethisfilebycallingthegenerate-definitionscommand)--validations-pathFILEPathpointingtothesetofvalidationstorun.Ifvalidations-pathandvalidationsarenotgiven,validaterunsallvalidations[envvar:VALIDATIONS_FILE]--validationsTEXTComma-separatedlistofvalidationstorun(e.g.--validationsRQ1,RQ2,RQ3).Ifvalidations-pathandvalidationsarenotgiven,validaterunsallvalidations[envvar:VALIDATIONS]--exit-on-failExitwithcode1whenvalidationsuccessisfalse.--yamlOutputyaml.--s3-endpoint-no-protocolTEXTEndpointforthes3servicewithoutprotocol[envvar:S3_ENDPOINT_NO_PROTOCOL]--s3-access-keyTEXTAccesskeyforthes3service[envvar:S3_ACCESS_KEY]--s3-secret-keyTEXTSecretkeyforthes3service[envvar:S3_SECRET_KEY]--s3-bucketTEXTBucketwherethegeopackageisonthes3service[envvar:S3_BUCKET]--s3-keyTEXTKeywherethegeopackageisinthebucket[envvar:S3_KEY]--s3-secureBOOLEANUseasecureTLSconnectionforS3.[envvar:S3_SECURE]--s3-virtual-hostingTEXTTRUEvalue,identifiesthebucketviaavirtualbuckethostname,e.g.:mybucket.cname.domain.com-FALSEvalue,identifiesthebucketasthetop-leveldirectoryintheURI,e.g.:cname.domain.com/mybucket.Convenienceparameter,sameasgdalAWS_VIRTUAL_HOSTING.[envvar:S3_VIRTUAL_HOSTING]--s3-signing-regionTEXTS3signingregion.Convenienceparameter,sameasgdalAWS_DEFAULT_REGION.[envvar:S3_SIGNING_REGION]--s3-no-sign-requestTEXTWhenset,requestsigningisdisabled.Thisoptionmightbeusedforbucketswithpublicaccessrights.Convenienceparameter,sameasgdalAWS_NO_SIGN_REQUEST.[envvar:S3_NO_SIGN_REQUEST]-v,--verbosityLVLEitherCRITICAL,ERROR,WARNING,INFOorDEBUG--helpShowthismessageandexit.Examples:dockerrun-v${PWD}:/gpkg--rmpdok/geopackage-validatorvalidate-t/path/to/generated_definitions.json--gpkg-path/gpkg/tests/data/test_allcorrect.gpkgRun with specific validations onlySpecified in file:dockerrun-v${PWD}:/gpkg--rmpdok/geopackage-validatorvalidate--gpkg-pathtests/data/test_allcorrect.gpkg--validations-pathtests/validationsets/example-validation-set.jsonOr specified on command line:dockerrun-v${PWD}:/gpkg--rmpdok/geopackage-validatorvalidate--gpkg-pathtests/data/test_allcorrect.gpkg--validationsRQ1,RQ2,RQ3Show validationsShow all the possible validations that are executed in the validate command.Usage:geopackage-validatorshow-validations[OPTIONS]Showallthepossiblevalidationsthatareexecutedinthevalidatecommand.

Options:-v,--verbosityLVLEitherCRITICAL,ERROR,WARNING,INFOorDEBUG--helpShowthismessageandexit.Generate table definitionsUsage:geopackage-validatorgenerate-definitions[OPTIONS]GeneratetabledefinitionforageopackageonlocalorS3storage.UsethegenerateddefinitionJSONorYAMLinthevalidationstepbyprovidingthetabledefinitionswiththe--table-definitions-pathparameter.Whenthefilepathisprecededwith'/vsi'thegdalvirtualfilesystemmethodwillbeusedtoaccessthefileonS3andwillnotbedirectlydownloaded.Seehttps://gdal.org/user/virtual_file_systems.htmlforfurtherexplanation.ForconveniencethegdalvsienvironmentparametersandoptionalparametersareprovidedwithanS3_insteadofanAWS_prefix.TheAWS_environmentparameterswillalsowork.Examples:viscurl:geopackage-validatorvalidate--gpkg-path/vsicurl/http://minio-url.nl/bucketname/key/to/public.gpkgvsis3:geopackage-validatorgenerate-definitions--gpkg-path/vsis3/bucketname/key/to/public.gpkg--s3-signing-regioneu-central-1--s3-secret-keysecret--s3-access-keyacces-key--s3-secure=false--s3-virtual-hostingfalse--s3-endpoint-no-protocolminio-url.nlS3_SECRET_KEY=secretS3_ACCESS_KEY=acces-keyS3_SIGNING_REGION=eu-central-1S3_SECURE=falseS3_VIRTUAL_HOSTING=falseS3_ENDPOINT_NO_PROTOCOL=minio-url.nlgeopackage-validatorgenerate-definitions--gpkg-path/vsis3/bucketname/key/to/public.gpkgAWS_SECRET_ACCESS_KEY=secretAWS_ACCESS_KEY_ID=acces-keyAWS_DEFAULT_REGION=eu-central-1AWS_HTTPS=NOAWS_VIRTUAL_HOSTING=FALSEAWS_S3_ENDPOINT=minio-url.nlgeopackage-validatorgenerate-definitions--gpkg-path/vsis3/bucketname/key/to/public.gpkg

Options:--gpkg-pathFILEPathpointingtothegeopackage.gpkgfile[envvar:GPKG_PATH]--yamlOutputyaml--s3-endpoint-no-protocolTEXTEndpointforthes3servicewithoutprotocol[envvar:S3_ENDPOINT_NO_PROTOCOL]--s3-access-keyTEXTAccesskeyforthes3service[envvar:S3_ACCESS_KEY]--s3-secret-keyTEXTSecretkeyforthes3service[envvar:S3_SECRET_KEY]--s3-bucketTEXTBucketwherethegeopackageisonthes3service[envvar:S3_BUCKET]--s3-keyTEXTKeywherethegeopackageisinthebucket[envvar:S3_KEY]--s3-secureBOOLEANUseasecureTLSconnectionforS3.[envvar:S3_SECURE]--s3-virtual-hostingTEXTTRUEvalue,identifiesthebucketviaavirtualbuckethostname,e.g.:mybucket.cname.domain.com-FALSEvalue,identifiesthebucketasthetop-leveldirectoryintheURI,e.g.:cname.domain.com/mybucket.Convenienceparameter,sameasgdalAWS_VIRTUAL_HOSTING.[envvar:S3_VIRTUAL_HOSTING]--s3-signing-regionTEXTS3signingregion.Convenienceparameter,sameasgdalAWS_DEFAULT_REGION.[envvar:S3_SIGNING_REGION]--s3-no-sign-requestTEXTWhenset,requestsigningisdisabled.Thisoptionmightbeusedforbucketswithpublicaccessrights.Convenienceparameter,sameasgdalAWS_NO_SIGN_REQUEST.[envvar:S3_NO_SIGN_REQUEST]-v,--verbosityLVLEitherCRITICAL,ERROR,WARNING,INFOorDEBUG--helpShowthismessageandexit.Local developmentWe advise using docker-compose for local development. This allows live editing and testing code with the correct gdal/ogr version with spatialite 5.0.0.
First build the local image with your machines user id and group id:docker-composebuild--build-argUSER_ID=`id-u`--build-argGROUP_ID=`id-g`UsageThere will be a script you can run like this:docker-composerun--rmvalidatorgeopackage-validatorThis command has direct access to the files found in this directory. In case you want
to point the docker-compose to other files, you can add or edit the volumes in thedocker-compose.yamlPython consoleIpython is available in the docker:docker-composerun--rmvalidatoripythonCode styleIn order to get nicely formatted python files without having to spend manual
work on it, run the following command periodically:docker-composerun--rmvalidatorblack.TestsRun the tests regularly. This also checks with pyflakes and black:docker-composerun--rmvalidatorpytestReleasingRelease in github by creating a new release in github.