Package: bas-web-map-inventory
MAGIC Web Map InventoryInventory of geospatial layers and web maps provided by the BAS Mapping and Geographic Information Centre (MAGIC),
visualised in Airtable.view the inventory in Airtable (Internal)See theData modelsection for more information about what this inventory holds.Note:This project is designed for internal use within MAGIC, but is offered to anyone with similar needs.UsageThis project runs in a container. See theSetupsection for setup instructions.If running on the BAS central workstations:$sshgeoweb@bslws01.nerc-bas.ac.uk
$web-map-inventory[task]Configuration, logs and data output are stored in/users/geoweb/.config/web-map-inventory/.If any errors occur they will be reported to Sentry and relevant individials alerted by email.CommandsversionReports the current application version.data fetchFetches information about servers, namespaces, repositories, styles, layers and layer groups from servers defined in a
data sources file. Fetched information is saved to an output data file.Options:-s,--data-sources-file-path:path to a data sources filedefault:data/sources.json-d,--data-output-file-path:path to a data sources filedefault:data/data.jsonNote:Currently this task results in new IDs being generated for each resource, even if it already exists. This will
lead to resources being removed and re-added unnecessarily but will always remain internally consistent.data validateValidates protocols offered by servers defined in a data sources file (by defaultdata/sources.json).Options:-s,--data-sources-file-path:path to a data sources filedefault:data/sources.json-i,--data-source-identifier:identifier of a server in the data sources fileuse special valueallto select all data sources-p,--validation-protocol:protocol to validatedefault:wmsNote:Currently this task is limited to the WMS (OGC Web Map Service) protocol.airtable statusChecks local items against Airtable to check whether they are up-to-date (current), outdated, missing or orphaned.airtable syncCreates, updates or removes items in Airtable to match local items.airtable resetRemoves all data from Airtable.Managing data sourcesEach data source is represented as an object in theserverlist indata/sources.json[1]. The structure of each
source depends on its type. For more general information, see theData sourcessection.[1] This file is either in the runtime path created duringSetupor~/.config/web-map-inventory/on the BAS
central servers).Adding new data sourcesNote:SeeSupported data sourcesfor currently supported data sources.Once added use thedata fetchtask.Adding aGeoServerdata sourcePropertyRequiredData TypeAllowed ValuesExample ValueDescriptionNotesidYesStringAULID(Universally Unique Lexicographically Sortable Identifier)01DRS53XAJNH0TNBW5161B6EWJUnique identifier for server/sourceSee below for how to generatelabelYesStringAny combination ofa-Z,A-Z,0-9,-,_a-1_AUsing a short, well-known identifier-hostnameYesStringAny valid hostnameexample.com--typeYesStringgeoserverSee allowed value--portYesStringAny valid port number8080-Usually80or8080api-pathYesString/geoserver/restSee allowed valueDefined by GeoServer-wms-pathYesString/geoserver/ows?service=wms&version=1.3.0&request=GetCapabilitiesSee allowed valueDefined by GeoServer-wfs-pathYesString/geoserver/ows?service=wfs&version=2.0.0&request=GetCapabilitiesSee allowed valueDefined by GeoServer-usernameYesStringAny valid GeoServer usernameadminUsually the GeoServer admin user-passwordYesStringPassword for GeoServer userpasswordUsually the GeoServer admin user-Note:Useulidgenerator.comto generate ULIDs manually.Example:{"id":"xxx","label":"example","hostname":"example.com","type":"geoserver","port":"80","api-path":"/geoserver/rest","wms-path":"/geoserver/ows?service=wms&version=1.3.0&request=GetCapabilities","wfs-path":"/geoserver/ows?service=wfs&version=2.0.0&request=GetCapabilities","username":"admin","password":"password"}Note:Make sure to include a comma between each source (but don't include a trailing comma) - e.g.:{"servers":[{"id":"xxx","label":"existing-server"...},{"id":"xxx","label":"new-server"...}]}ImplementationFlask application using theairtable-python-wrapperlibrary to
interact with the Airtable API.AirtableData is synced to theMAGIC Maps and Layers InventoryBase in theBAS MAGICWorkspace.Data modelThis project, an inventory, consists of information held in geospatial services. The data model is intended to be
generic to support different data sources and technologies.This data model consists of:Servers: Represent a source of geospatial information, such as an instance of a technology or a whole platformNamespaces: Represent a logical grouping of resources within a server/endpointRepositories: Represent a data source that backs one or more layersStyles: Represent a definition for how data in a layer should be represented/presentedLayers: Represent a logical unit of geospatial informationLayer Groups: Represent a logical grouping of one or more layers that should be treated as a single, indivisible
unitAJSON Schemadescribes this schema. It is used
internally for validating data prior to use but is also published for use by others if needed as:data-schema-v1.json.Data sourcesData sources areserversin the projectData modeland define connection details for APIs and services
each server type provides for fetching information about components they contain (e.g. listinglayers).A data sources file,data/sources.json, is used for recording these details. An example is available indata/sources.example.json. See theAdding a data sourcesection for more information.A JSON Schema,bas_web_map_inventory/resources/json_schemas/data-sources-schema.json, validates this file.Supported data sourcesGeoServerUsing a combination of its admin API and WMS/WFS OGC endpointsConfigurationConfiguration options are set withinbas_web_map_inventory/config.py.AllOptionsare defined in aConfigbase class, with per-environment sub-classes overriding
and extending these options as needed. The active configuration is set using theFLASK_ENVenvironment variable.Most options can beSet using environment variables or files.Configuration optionsOptionRequiredEnvironmentsData Type (Cast)SourceAllowed ValuesDefault ValueExample ValueDescriptionNotesFLASK_APPYesAllString.flaskenvValidFLASK_APPvaluemanage.pySee default valueSeeFlask documentation-APP_ENABLE_SENTRYNoAllBoolean.flaskenvTrue/FalseFalse(fordevelopment/testing),True(forstaging/production)TrueFeature flag forError reporting-APP_ENABLE_FILE_LOGGINGNoAllBoolean.flaskenvTrue/FalseFalseFalseFeature flag for writingApplication Logsto a file in addition to standard out-SENTEY_DSNYesYesString.flaskenvSentry DSNfor this projecthttps://c69a62ee2262460f9bc79c4048ba764f@sentry.io/1832548See default valueSentryData Source NameThis value is not a secretAPP_LOG_FILE_PATHNoAllString.flaskenvValid file path/var/log/app/app.log/var/log/app/app.logPath to application log file, if enabled-AIRTABLE_API_KEYYesAllString.envValidAirTable API key-keyxxxxxxxxxxxxxxAirTable API Key-AIRTABLE_BASE_IDYesAllString.envValidAirTable Base ID-appxxxxxxxxxxxxxxID of the AirTable Base to populate/use-Options are set as strings and then cast to the data type listed above. SeeEnvironment variablesfor information about an options 'Source'.Flask also has a number ofbuiltin configuration options.Setting configuration optionsVariable configuration options can be set using environment variables or environment files:SourcePriorityPurposeNotesOS environment variables1stGeneral/Runtime-.env2ndSecret/private variablesGenerate by copying.env.example.flaskenv3rdNon-secret/public variablesGenerate by copying.flaskenv.exampleNote:these sources are aFlask convention.Error trackingErrors in this service are tracked with Sentry:Sentry dashboardGitLab dashboardError tracking will be enabled or disabled depending on the environment. It can be manually controlled by setting theAPP_ENABLE_SENTRYvariable in.flaskenv.LoggingLogs for this service are written tostdoutand a log file,/var/log/app/app.py, depending on the environment.File based logging can be manually controlled by setting theAPP_ENABLE_FILE_LOGGINGandAPP_LOG_FILE_PATHvariables in.flaskenv.Note:IfAPP_LOG_FILE_PATHis changed, the user the container runs as must be granted suitable write permissions.XML CatalogueAnXML Catalogis used to cache XML files locally (typically XSD's for
schemas). This drastically speeds up XML parsing and removes a dependency on remote endpoints.XML files in the catalogue are typically stored inbas_web_map_inventory/resources/xml_schemas/.Different catalogue files are used for different container variants due to differences in the applications location::latest:./support/xml-schemas/catalogue.xml/deploy:provisioning/docker/catalog.xmlIn either case, the catalogue is available within the container at the conventional path,/etc/xml/catalog, and will
be used automatically by most XML libraries and tools (such aslxmlandxmllint).SetupThe application for this project runs as a Docker container.Once setup, see theData sourcesandUsagesections for how to use and run the
application.Note:This project can run locally or on the BAS central workstations using Podman. You will need
access to the private BAS Docker Registry (part ofgitlab.data.bas.ac.uk) and for IT to enable Podman in your user account. Unless noted,dockercommands listed here can be replaced withpodman.$dockerlogindocker-registry.data.bas.ac.uk
$dockerpulldocker-registry.data.bas.ac.uk/magic/web-map-inventory/deploy:stableNote:Other image tagsare available if
you want to run pre-release versions, or a specific, previous, version.Before you can run the container, you will need to create a runtime directory that will live outside of the container.
You will need to create the requiredConfiguration files. Any generated output will also be saved
here.$mkdir-p~/.config/web-map-inventoryOptional wrapper scriptIf using podman, a wrapper script,support/container-wrapper/podman-wrapper.sh, is available to make running the
container easier.To use, copy this script and enable it to be executed:$mkdir~/bin# copy `support/container-wrapper/podman-wrapper.sh` as `~/bin/web-map-inventory`$chmod+x~/bin/web-map-inventoryThen ensure~/binis part of the user's path (useecho $PATHto check), if it isn't edit the user's shell to include
it (these instructions assume the bash shell and the absolute path to the user's home directory is/home/foo):$vi~/.bash_rc# add `export PATH="/home/foo/bin:$PATH" then save the file and reload the user's shellYou should now be able to runweb-map-inventoryto run the container.TerraformTerraform is used to provision resources required to allow JSON Schemas for data resources and data sources to be
accessed externally.Access to theBAS AWS accountis needed to provisioning these resources.Note:This provisioning should have already been performed (and applies globally). If changes are made to this
provisioning it only needs to be applied once.# start terraform inside a docker container$cdprovisioning/terraform
$docker-composerunterraform# setup terraform$terraforminit# apply changes$terraformvalidate
$terraformfmt
$terraformapply# exit container$exit$docker-composedownTerraform remote stateState information for this project is stored remotely using aBackend.Specifically theAWS S3backend as part of theBAS Terraform Remote Stateproject.Remote state storage will be automatically initialised when runningterraform init. Any changes to remote state will
be automatically saved to the remote backend, there is no need to push or pull changes.Remote state authenticationPermission to read and/or write remote state information for this project is restricted to authorised users. Contact
theBAS Web & Applications Teamto request access.See theBAS Terraform Remote Stateproject for how these
permissions to remote state are enforced.Development$gitclonehttps://gitlab.data.bas.ac.uk/MAGIC/web-map-inventory
$cdmap-layer-indexDevelopment environmentThe:latestcontainer image is used for developing this project. It can run locally using Docker and Docker Compose:$dockerlogindocker-registry.data.bas.ac.uk
$docker-composepullappThen create/configure requiredConfiguration files:$cp.env.example.env
$cp.flaskenv.example.flaskenv
$cpdata/sources.example.jsondata/sources.jsonTo run/test application commands:$docker-composerunappflask[task][1] You will need access to the private BAS Docker Registry (part ofgitlab.data.bas.ac.uk) to pull this image. If you don't, you can build the relevant
image/tag locally instead.Code StylePEP-8 style and formatting guidelines must be used for this project, with the exception of the 80 character line limit.Blackis used to ensure compliance, configured inpyproject.toml.Black can beintegratedwith a
range of editors, such as PyCharm, to perform formatting automatically.To apply formatting manually:$docker-composerunappblackbas_web_map_inventory/To check compliance manually:$docker-composerunappblack--checkbas_web_map_inventory/Checks are ran automatically inContinuous Integration.Python versionWhen upgrading to a new version of Python, ensure the following are also checked and updated where needed:Dockerfile:base stage image (e.g.FROM python:3.X-alpine as basetoFROM python:3.Y-alpine as base)pre-compiled wheels (e.g.https://.../linux_x86_64/cp3Xm/lxml-4.5.0-cp3X-cp3X-linux_x86_64.whltohttp://.../linux_x86_64/cp3Ym/lxml-4.5.0-cp3Y-cp3Y-linux_x86_64.whl)provisioning/docker/Dockerfile:base stage image (e.g.FROM python:3.X-alpine as basetoFROM python:3.Y-alpine as base)pre-compiled wheels (e.g.http://.../linux_x86_64/cp3Xm/lxml-4.5.0-cp3X-cp3X-linux_x86_64.whltohttp://.../linux_x86_64/cp3Ym/lxml-4.5.0-cp3Y-cp3Y-linux_x86_64.whl)provisioning/docker/catalog.xml:update the path to the Python package (e.g.file://.../lib/python3.X/site-packages/...tofile://.../lib/python3.Y/site-packages/...)pyproject.toml[tool.poetry.dependencies]python(e.g.python = "^3.X"topython = "^3.Y")[tool.black]target-version(e.g.target-version = ['py3X']totarget-version = ['py3Y'])DependenciesPython dependencies for this project are managed withPoetryinpyproject.toml.The development container image installs both runtime and development dependencies. Deployment images only install
runtime dependencies.Non-code files, such as static files, can also be included in thePython packageusing theincludekey inpyproject.toml.To add a new (development) dependency:$docker-composerunappash
$poetryadd[dependency](--dev)Then rebuild the development container and push to GitLab (GitLab will rebuild other images automatically as needed):$docker-composebuildapp
$docker-composepushappStatic security scanningTo ensure the security of this API, source code is checked againstBanditfor issues
such as not sanitising user inputs or using weak cryptography. Bandit is configured in.bandit.Warning:Bandit is a static analysis tool and can't check for issues that are only be detectable when running the
application. As with all security tools, Bandit is an aid for spotting common mistakes, not a guarantee of secure code.To run checks manually:$docker-composerunappbandit-r.Checks are ran automatically inContinuous Integration.LoggingUse the Flask default logger. For example:app.logger.info('Log message')When outside of a route/command usecurrent_app:fromflaskimportcurrent_appcurrent_app.logger.info('Log message')File pathsUse Python'spathliblibrary for managing file paths.Where displaying a file path to the user ensure it is always absolute to aid in debugging:frompathlibimportPathfoo_path=Path("foo.txt")print(f"foo_path is:{str(foo_path.absolute())}")JSON SchemasJSON Schema's can be developed usingjsonschemavalidator.net.XML Catalogue additionsIf new functionality is added that depends on XML files, such as XSDs, it isstronglyrecommended to add them to theXML catalogue, especially where they are used in tests.Once added, you will need to rebuild and push the project Docker image (see theDependenciessection
for more information).Editor supportPyCharmA run/debug configuration,App, is included in the project.TestingAll code in thebas_web_map_inventorymodule must be covered by tests, defined intests/. This project usesPyTestwhich should be ran in a random order usingpytest-random-order.To run tests manually from the command line:$docker-composerunapp-eFLASK_ENV=testingapppytest--random-orderTo run tests manually using PyCharm, use the includedApp (Integration)run/debug configuration.Tests are ran automatically inContinuous Integration.Test coveragepytest-covis used to measure test coverage.To prevent noise,.coveragercis used to omit empty__init__.pyfiles from reports.To measure coverage manually:$docker-composerun-eFLASK_ENV=testingapppytest--cov=bas_web_map_inventory--cov-fail-under=100--cov-report=html.Continuous Integrationwill check coverage automatically and fail if less than 100%.Continuous IntegrationAll commits will trigger a Continuous Integration process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.DeploymentPython packageThis project is distributed as a Python package, hosted inPyPi.Source and binary packages are built and published automatically usingPoetryinContinuous Delivery.Package versions are determined automatically using thesupport/python-packaging/parse_version.pyscript.Docker imageThe projectPython packageis available as a Docker/OCI image, hosted in the private BAS Docker
Registry (part ofgitlab.data.bas.ac.uk).Continuous Deliverywill automatically:build a/deploy:latestimage for commits to themasterbranchbuild a/deploy:release-stableand/deploy:release-[release]image for tagsdeploy new images to the BAS central workstations (by runningpodman pull [image]from the workstations)Continuous DeploymentAll commits will trigger a Continuous Deployment process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.Release procedureFor all releases:create a release branchclose release inCHANGELOG.mdpush changes, merge the release branch intomasterand tag with versionFeedbackThe maintainer of this project is the BAS Mapping and Geographic Information Centre (MAGIC), they can be contacted at:servicedesk@bas.ac.uk.Issue trackingThis project uses issue tracking, see theIssue trackerfor more information.Note:Read & write access to this issue tracker is restricted. Contact the project maintainer to request access.License© UK Research and Innovation (UKRI), 2019 - 2020, British Antarctic Survey.You may use and re-use this software and associated documentation files free of charge in any format or medium, under
the terms of the Open Government Licence v3.0.You may obtain a copy of the Open Government Licence athttp://www.nationalarchives.gov.uk/doc/open-government-licence/