Package: scar-add-metadata-toolbox
SCAR Antarctic Digital Database (ADD) Metadata ToolboxEditor, repository and data catalogue forSCAR Antarctic Digital Database (ADD) discovery metadata.StatusThis project is a mature alpha.This means the core components needed have now been implemented but are subject to considerable change and refactoring.Between releases major parts of this project may be replaced whilst the project evolves. As major non-core features are
yet to be implemented the shape and scope of this project may change significantly. It is still expected that this
project will grow to cover other MAGIC datasets and products in future, and more widely to act as the seed for a new
BAS wide data catalogue.The0.2.0release has effectively been a complete rewrite of the project to reorganise and reimplement prototype code
in a more structured way. Automated integration tests have been added and the project is now open-sourced.Some undesirable code from the0.1.0release still remains, to workaround issues in other packages until they can be
properly addressed. This code has been moved to a 'Hazardous Materials' (scar_add_metadata_toolbox.hazmat) module.Further information on upcoming changes to this project can be found in the issues and milestones inGitLab (internal).Note:This project is designed to meet an internal need within theMapping and Geographic Information Centre (MAGIC)at BAS. It has been open-sourced
in case it's of use to others with similar needs.OverviewThis project is comprised of several components:metadata editor - for maintaining metadata records written as JSON files via theBAS Metadata Libraryunpublished (working) metadata repository - using an embedded, authenticated PyCSW cataloguepublished metadata repository - using an embedded, partially-authenticated PyCSW cataloguedata catalogue - using a static websiteThese components map to components 2, 4 and 6 in the draft ADD data workflow
(#139 (internal)).Metadata records are persisted within the themetadata repositories, using the ISO 19115-2 (geographic information)
metadata standard.Access to unpublished records, and the ability to publish/retract records is restricted to restricted to relevant ADD
project members. Once published, records can viewed through thedata catalogue, which presents them as human readable
items, (such as visualising geographic extents on a map). Manually curated collections provide a means to group items.Thedata catalogueis rendered as a static website for improved performance, reliability and ease of hosting.
Currently this content is accessed within the existingBAS data catalogue.UsageMetadata editorThemetadata editorcomponent of this project is ran on the BAS central workstations using the shared MAGIC user:$sshgeoweb@bslws01.nerc-bas.ac.uk
$scar-add-metadata-toolbox[command]The editor is configured using a settings file:/users/geoweb/.config/scar-add-metadata-toolbox/.env.Metadata repositories and data catalogueTheunpublished repository,published repositoryanddata cataloguerun as aservicein the experimentalMAGIC Nomad cluster.Any errors will be automatically reported toSentryand relevant individuals alerted by email.Workflowsadding new recordsupdating existing recordsadding new collectionsupdating existing collectionsAvailable commandsCommand line referenceImplementationFlask application usingCSWto storeMetadata recordsand display them asItemsinCollectionsrendered asJinja templatesserved as astatic websitewithin theBAS data catalogue.CSW catalogues are backed by PostGIS databases, secured usingOAuth. Contact forms for feedback and items in
the static catalogue useMicrosoft Power Automate. Legal policies use templates from theLegal Policiesproject.ArchitectureThis diagram shows the main concepts in this project and how they relate:Metadata recordsMetadata records are the content and data within project. Records describe resources, which are typically datasets
within the ADD, e.g. a record might describe the Antarctic Coastline dataset. Records are based on the ISO 19115
metadata standard (specifically 19115-2:2009), which defines an information model for geographic data.Records are stored/persisted in a records repository (implemented usingCSW) or in files for import and export.A metadata record includes information to answer questions such as:what is this dataset?what formats is this dataset available in?what projection does this dataset use?what keywords describe the themes, places, etc. related to this dataset?why is this dataset useful?who is this dataset for?who made this dataset?who can I contact with any questions about this dataset?when was this dataset created?when was it last updated?what time period does it cover?where does this dataset cover?how was this dataset created?how can trust the quality of this dataset?how can I download or access this dataset?This metadata is termed 'discovery metadata' (to separate it from metadata for calibration or analysis for example). It
helps users find metadata in catalogues or search engines, and then to help them decide if the data is useful to them.The information in a metadata record is encoded in a different formats at different stages:during editing, records are encoded as JSON, using theBAS Metadata Library's record configurationwhen stored in a repository, records are encoded as XML using the ISO 19139 standardwhen viewed in the data catalogue, records are encoded in HTML or as (styled) XMLThese different formats are used for different reasons:JSON is easily understood by machines and is concise to understand and edit by humansXML is also machine readable but more suited/mature for complex standards such ISO 19139HTML is designed for presenting information to humans, with very flexible formatting optionsItemsItems representMetadata recordsbut in a form intended for human consumption. They are derived
fromRecordsand are specific to the data catalogue, allowing greater flexibility compared to the
strict rigidity and formality enforced by metadata records.For example, a resource's coordinate reference system may be defined asurn:ogc:def:crs:EPSG::3031in a metadata
record but will be shown asWGS 84 / Antarctic Polar Stereographic (EPSG:3031)in items. Both are technically correct
but the descriptive version is easier for a human to understand, at the sake of being less precise and harder for
machines to parse.As items are derived from records, they are not persisted themselves, except as rendered pages within the data catalogue
static site.CollectionsCollections are a simple way to groupItemstogether based on a shared purpose, theme or topic. They are
specific to the data catalogue and are not based on metadata records.Collections are stored/persisted in a collections repository (implemented as a JSON file) or in files for import and
export.They support a limited set of properties compared to records/items:PropertyData TypeRequiredDescriptionidentifierStringYesUUIDtitleStringYesDescriptive titletopicsArrayYesBAS Research Topics associated with collectiontopics.*StringYesBAS Research TopicpublishersArrayYesData catalogue publishers associated with collectionpublishers.*StringYesData catalogue publishersummaryStringYesDescriptive summary, supports markdown formattingitem_identifiersArrayYesItems associated with collection, specified by their Item IDitem_identifiers.*StringYesItem IDNote:Items in collections will be shown in the order they are listed in theitem_identifiersproperty.For example:{"identifier":"1790c9d5-af77-4a03-9a08-6ba8e83ce748","title":"Operation Tabarin","topics":["Living and Working in Antarctica"],"publishers":["BAS Archives Service"],"summary":"A secret British Antarctic expedition launched in 1943 during the course of World War II ...","item_identifiers":["82f3fe32-6d6b-4e7a-8256-690ce99fc653","88a22198-36e0-4aff-9099-aae1dfd7baa9","35c6d732-3acc-4044-9c8f-680eed39268a"]}OAuthThis project uses OAuth to protect access to theUnpublishedandPublishedrepositories via theMicrosoft (Azure) identity platform.TheUnpublishedandPublishedrepositories are registered together as the resource to be protected with different
scopes and roles to authorise users to read, write and publish records. TheFlask Azure AD OAuth Provideris used to verify access tokens when CSW
requests are made and enforce these permissions as needed.TheMetadata editoris registered as a separate application as a client that will interact with protected resource
(i.e. to read, write and publish records). TheMicrosoft Authentication Library (MSAL) for Pythonlibrary is used to request access tokens using the OAuth device code grant type.Both applications are registered in the NERC Azure tenancy administered by theUKRI/NERC DDaTteam, currently via the oldRTS Helpdesk.TheAzure Portalis used to assign permissions to applications and users as needed:assigning permissions to usersCSWTheUnpublishedandPublishedrepositories are implemented as embeddedPyCSWservers. The
embedded mode allowing integration with Flask for authentication and authorisation of requests viaOAuth.The CSW transactional profile is used extensively for clients (such as theMetadata editorandData catalogue) to
insert, update and delete records programmatically.The CSW version is fixed to2.0.2because it's the latest version supported byOWSLib, the CSW client used by theMetadata editor.Note:The CSW repositories are considered to be APIs, and so ran as services through theBAS API Load Balancer(internal) with documentation in theBAS API Documentationproject (internal).Note:Some elements of both the PyCSW server and the OWSLib client have been extended by this project to incorporate
OAuth support. These modifications will be formalised, ideally as upstream contributions, but currently reside within
the 'Hazardous Materials' module as a number of unsightly workarounds are currently needed.Max records limitBoth PyCSW (CSW servers) and OWSLib (CSW clients) have a maximum record of 100 per request.CSW backing databasesCSW servers are backed using PostGIS (PostgreSQL) databases provided by BAS IT (via the central Postgres databasebsldb). As PyCSW uses a single table for all records, all servers share the same database and schema, configured
through SQLAlchemy connection strings.Separate databases are used for each environment (development, staging and production). Credentials are stored in the
MAGIC 1Password shared vault. In local development, a local PostGIS database configured indocker-compose.ymlcan be
used:postgresql://postgres:password@csw-db/postgres`To test against real data in a non-production environment, use the staging environment database, which is synced from
the production database automatically by BAS IT every Tuesday at 02:00.Jinja templatesA series ofJinja2templates are used for rendering pages in theData catalogue.Templates use theBAS Style Kit Jinja Templatesand styled
using theBAS Style Kit.S3 static websiteRendered pages and other assets are hosted through an AWS S3 bucket with static website hosting enabled.Reverse proxy rules are used to expose content from this static site within the existing/legacy BAS Data Catalogue, DMS.Feedback and contact formsA MicrosoftPower AutomateFlow is used to process feedback and contact form submissions. Messages support Markdown formatting, converted to HTML
prior to submission. On submitted, Power Automate creates an issue for the message in a relevant GitLab project.Sentry error trackingErrors in this service are tracked with Sentry:Sentry dashboardGitLab dashboardError tracking will be enabled or disabled depending on the environment. It can be manually controlled by setting theAPP_ENABLE_SENTRYConfiguration option.Application loggingLogs for this service are written tostdout/stderras appropriate.ConfigurationApplication configuration options are set in per-environment classes extending a baseConfigclass inscar_add_metadata_toolbox/config.py. The active environment is set using theFLASK_ENVenvironment variable.Configuration options are defined, and documented, using class properties. Some configuration options may optionally be
set at runtime using environment variables. If not set, default values will be used.Configuration OptionDescriptionAllowed ValuesExample ValueAPP_ENABLE_SENTRYFeature flag to enable/disable Sentry error trackingtrue/falsetrueAPP_LOGGING_LEVELMinimum logging level to include in application logsdebug/info/warning/error/criticalwarningAPP_AUTH_SESSION_FILE_PATHPath to file used for authentication informationvalid file path/home/user/.config/scar_add_metadata_toolbox/auth.jsonAPP_COLLECTIONS_PATHPath to file used for data catalogue collectionsvalid file path/home/user/.config/scar_add_metadata_toolbox/collections.jsonAPP_SITE_PATHPath to directory used for rendered static site contentvalid directory path/home/user/.config/scar_add_metadata_toolbox/_siteCSW_ENDPOINT_UNPUBLISHEDCSW endpoint for accessing unpublished cataloguevalid URLhttp://example.com/csw/unpublishedCSW_ENDPOINT_PUBLISHEDCSW endpoint for accessing published cataloguevalid URLhttp://example.com/csw/publishedCSW_SERVER_CONFIG_UNPUBLISHED_ENDPOINTEndpoint at which to run unpublished CSW cataloguevalid URLhttp://example.com/csw/unpublishedCSW_SERVER_CONFIG_PUBLISHED_ENDPOINTEndpoint at which to run published CSW catalogueValid URLhttp://example.com/csw/publishedCSW_SERVER_CONFIG_UNPUBLISHED_DATABASE_CONNECTIONConnection string for unpublished CSW catalogue backing databaseValid SQLAlchemy connection stringpostgresql://postgres:password@db.example.com/postgresCSW_SERVER_CONFIG_PUBLISHED_DATABASE_CONNECTIONConnection string for published CSW catalogue backing databaseValid SQLAlchemy connection stringpostgresql://postgres:password@db.example.com/postgresAPP_S3_BUCKETAWS S3 bucket name used for hosting static website contentValid AWS S3 bucket nameadd-catalogue.data.bas.ac.ukThese options are typically set when running this application as a client (metadata editor and data catalogue):APP_LOGGING_LEVELAPP_AUTH_SESSION_FILE_PATHAPP_COLLECTIONS_PATHAPP_SITE_PATHCSW_ENDPOINT_UNPUBLISHEDCSW_ENDPOINT_PUBLISHEDAPP_S3_BUCKETThese options are typically set when running this application as a server (metadata repositories):APP_LOGGING_LEVELCSW_SERVER_CONFIG_UNPUBLISHED_ENDPOINTCSW_SERVER_CONFIG_PUBLISHED_ENDPOINTCSW_SERVER_CONFIG_UNPUBLISHED_DATABASE_CONNECTIONCSW_SERVER_CONFIG_PUBLISHED_DATABASE_CONNECTIONSetupContinuous deploymentwill configure this application to run on the BAS central workstations
as a Podman container, using an automatically generated launch script and environment variables.Continuous deploymentwill configure this application to run in the experimentalMAGIC Nomad cluster, using an automatically
generated job definition.See theUsagesection for how to use the application.PyCSW backing database setupBacking databases for PyCSW servers require initialisation using thecsw setupapplicationCLI commandfor both thepublishedandunpublishedrepositories.Note:Backing databases must use the PostgreSQL engine with the PostGIS extension enabled.Normally this command will create the required database table, geometry column and relevant indexes. As catalogues only
require a single table, multiple can be stored in the same database/schema. However, two of the indexes used
(fts_gin_idx[full text search] andwkb_geometry_idx[binary geometry]) are named non-uniquely, preventing multiple
catalogues being co-located in the same schema.This appears to be an oversight, as all other indexes are made unique by prefixing them with the name of the records
table, and doing this manually for these indexes appears to work without issue. To workaround this issue, you will need
to manually modify the indexes of catalogue tables once they've been setup.Assuming theUnpublished catalogueis setup first, perform these stepsbeforesetting up thePublished catalogue:verify that therecords_unpublishedtable was created successfully (containsfts_gin_idxandwkb_geometry_idxindexes)alter the affected indexes in therecords_unpublishedtable [1]setup thePublished catalogueflask csw setup publishedalter the affected indexes in the second table [2][1]ALTERINDEXfts_gin_idxRENAMETOix_records_unpublished_fts_gin_indx;ALTERINDEXwkb_geometry_idxRENAMETOix_unpublished_wkb_geometry_idx;[2]ALTERINDEXfts_gin_idxRENAMETOix_records_published_fts_gin_indx;ALTERINDEXwkb_geometry_idxRENAMETOix_published_wkb_geometry_idx;Azure permissionsTerraformwill create and configure the relevant Azure application registrations required for usingOAuthto protect the CSW catalogues. However manual approval by a Tenancy Administrator is needed to grant
the registration representing theclientrole of the application access to the registration for theserverrole.This has been approved by NERC RTS in#3.TerraformTerraform is used for:resources required for hosting theData cataloguecomponent as a static websiteresources required for protecting and accessing theunpublished repository,published repositorycomponentsa templated job file for Nomad duringContinuous deploymenta templated launch script for Podman duringContinuous deploymentAccess to theBAS AWS account,Terraform remote stateand NERC Azure tenancy are required to provision these resources.Note:The templated Podman and Nomad runtime files are not included in Terraform state.$cdprovisioning/terraform
$dockercomposerunterraform

$azlogin--allow-no-subscriptions

$terraforminit
$terraformvalidate
$terraformfmt
$terraformapply

$exit$dockercomposedownOnce provisioned the following steps need to be taken manually:set branding icons (if desired)setAzure permissionsassign rolesto users and/or groupssetaccessTokenAcceptedVersion: 2in both application registration manifestsNote:Assignments are 1:1 between users/groups and roles but there can be multiple assignments. I.e. rolesFooandBarcan be assigned to the same user/group by creating two role assignments.Terraform remote stateState information for this project is stored remotely using aBackend.Specifically theAWS S3backend as part of theBAS Terraform Remote Stateproject.Remote state storage will be automatically initialised when runningterraform init. Any changes to remote state will
be automatically saved to the remote backend, there is no need to push or pull changes.Remote state authenticationPermission to read and/or write remote state information for this project is restricted to authorised users. Contact
theBAS Web & Applications Teamto request access.See theBAS Terraform Remote Stateproject for how these
permissions to remote state are enforced.Docker image tag expiration policyThe Docker image for this project uses aTag expiration policywhich needs to be
configured manually inGitLab.Expiration policy:enabledExpiration interval:90 daysExpiration schedule:Every weekNumber of tags to retain:10 tag per image nameTags with names matching this regex pattern will expire:(review.+|build.+)Tags with names matching this regex pattern will be preserved:release.+BAS ITManually request a new PostGIS database for the CSW catalogue backing databases from the BAS IT ServiceDesk.Manually request a new application to be deployed from the BAS IT ServiceDesk using therequest template.See#44for an example.BAS API Load BalancerManuallyadd a new serviceand relateddocumentation.See#60for an example.Development$gitclonehttps://gitlab.data.bas.ac.uk/MAGIC/add-metadata-toolbox
$cdadd-metadata-toolboxDevelopment environmentA flexible development environment is available for developing this application locally. It can be used in a variety of
ways depending on what is being developed:all components, the Flask application, CSW database and static website can be ran locallyuseful for end-to-end testinguseful for testing changes to how data is loaded into the CSW cataloguesthe Flask application can be ran directly, without needing to convert it into a static siteuseful for iterating on changes to the data catalogue websitethe Flask application can use the production CSW databaseuseful for testing with real-world dataThe local development environment is defined using Docker Compose in./docker-compose.yml. It consists of:anappservice for running the Flask application as a web applicationanapp-cliservice for runningapplication commandsacsw-dbservice for storing data added to local CSW catalogues (if used)awebservice for serving a local version of the data catalogue static site (if used)To create a local development environment:pull docker images:docker compose pull[1]run the Docker Compose stack:docker compose upthe Flask application will be available directly at:http://localhost:9000the static site will be available at:http://localhost:9001run applicationCommands[2]To destroy a local development environment:rundocker compose down[1] This requires access to the BAS Docker Registry (part ofgitlab.data.bas.ac.uk):$dockerlogindocker-registry.data.bas.ac.ukNote:You will need to sign-in using your GitLab credentials (your password is set through your GitLab profile) the
first time this is used.[2] In a new terminal:$dockercomposerunapp-cliflask[task]Development containerA development container image, defined by./Dockerfile, is built manually, tagged as:latestand hosted in the
private BAS Docker Registry (part ofgitlab.data.bas.ac.uk):docker-registry.data.bas.ac.uk/magic/add-metadata-toolboxIt is separate to thedeployment containerand installs both runtime and developmentdependencies(deployment containers only install runtime dependencies).If you don't have access to the BAS Docker Register, you can build this image locally usingdocker compose build app.Python versionWhen upgrading to a new version of Python, ensure the following are also checked and updated where needed:Dockerfile:base stage image (e.g.FROM python:3.X-alpine as basetoFROM python:3.Y-alpine as base)pre-compiled wheels (e.g.https://.../linux_x86_64/cp3Xm/lxml-4.5.0-cp3X-cp3X-linux_x86_64.whltohttp://.../linux_x86_64/cp3Ym/lxml-4.5.0-cp3Y-cp3Y-linux_x86_64.whl)support/docker-packaging/Dockerfile:base stage image (e.g.FROM python:3.X-alpine as basetoFROM python:3.Y-alpine as base)pre-compiled wheels (e.g.http://.../linux_x86_64/cp3Xm/lxml-4.5.0-cp3X-cp3X-linux_x86_64.whltohttp://.../linux_x86_64/cp3Ym/lxml-4.5.0-cp3Y-cp3Y-linux_x86_64.whl)pyproject.toml[tool.poetry.dependencies]python(e.g.python = "^3.X"topython = "^3.Y")[tool.black]target-version(e.g.target-version = ['py3X']totarget-version = ['py3Y'])Package structureAll code for this project should be defined in thescar_add_metadata_toolboxpackage,
with the exception of tests.In brief, this package is comprised of these modules:scar_add_metadata_toolbox- containsFlask applicationscar_add_metadata_toolbox.classes- contains classes for concepts (Repositories, Records, Items, Collections)scar_add_metadata_toolbox.commands- contains Flask blueprints used for CLI commandsscar_add_metadata_toolbox.config- containsFlask configurationscar_add_metadata_toolbox.csw- contains classes forCSWservers and clientsscar_add_metadata_toolbox.hazmat- contains'Hazardous Material' codescar_add_metadata_toolbox.static- contains static site assets (CSS, JS, etc.)scar_add_metadata_toolbox.templates- containsApplication templatesscar_add_metadata_toolbox.utils- contains various utility/helper methods and classesHazardous Materials moduleWhilst this application has been developed, extended/modified versions of class from 3rd party packages have been
created to address unresolved bugs or add new required functionality. As this code often requires workarounds and hacks
it is ugly, non-standard and against established best practices (such as not to use mocks outside of tests).In time, these changes and additions are expected to be either incorporated into upstream packages, or if not possible,
into forked packages that we maintain. Until then, this code is kept in a 'Hazardous Materials' (Hazmat) module,scar_add_metadata_toolbox.hazmat, to indicate that it shouldn't be treated like other modules in this project.Ideally no additional code will be added to this module, however if other changes/extensions need to be made in a
non-clean way then they should be added here, rather than 'polluting' the main package.DependenciesPython dependencies for this project are managed withPoetryinpyproject.toml.Non-code files, such as static files, can also be included in thePython packageusing theincludekey inpyproject.toml.Adding new dependenciesTo add a new (development) dependency:$dockercomposerunappash
$poetryadd[dependency](--dev)Then rebuild theDevelopment containerand push to GitLab (GitLab will rebuild other images
automatically as needed):$dockercomposebuildapp
$dockercomposepushappUpdating dependencies$dockercomposerunappash
$poetryupdateThen rebuild theDevelopment containerand push to GitLab (GitLab will rebuild other images
automatically as needed):$dockercomposebuildapp
$dockercomposepushappStatic security scanningTo ensure the security of this API, source code is checked againstBanditfor issues
such as not sanitising user inputs or using weak cryptography. Bandit is configured in.bandit.Warning:Bandit is a static analysis tool and can't check for issues that are only be detectable when running the
application. As with all security tools, Bandit is an aid for spotting common mistakes, not a guarantee of secure code.To run checks manually:$dockercomposerunappbandit-r.Checks are ran automatically inContinuous Integration.Code StylePEP-8 style and formatting guidelines must be used for this project, with the exception of the 80 character line limit.Blackis used to ensure compliance, configured inpyproject.toml.Black can beintegratedwith a range of editors, such
as PyCharm, to perform formatting automatically.To apply formatting manually:$dockercomposerunappblackscar_add_metadata_toolbox/To check compliance manually:$dockercomposerunappblack--checkscar_add_metadata_toolbox/Checks are ran automatically inContinuous Integration.Flask applicationThe Flask application representing this project is defined in thescar_add_metadata_toolboxpackage. The application uses theapplication factorypattern.Flask Blueprints are used to logically organise application commands, currently all within thescar_add_metadata_toolbox.commandsmodule. Until this is refactored, additional commands should be registered in the
same module.Flask configurationThe Flask application's configuration (app.config) is populated from an environment specific class in thescar_add_metadata_toolbox.configmodule.New configuration options should be added to the base config class as properties, overridden as needed in environment
sub-classes. Where a configuration should be configurable at runtime it should be read as an environment variable and
documented in theConfigurationsection.LoggingUse the Flask application's logger, for example:fromflaskimportcurrent_appcurrent_app.logger.info('Log message')File pathsUse Python'spathliblibrary for file paths.Where displaying a file path to the user, use the absolute form to aid in debugging:frompathlibimportPathfoo_path=Path("foo.txt")print(f"foo_path is:{str(foo_path.absolute())}")TemplatesApplication templates use the Flask application's Jinja environment configured to use general templates from theBAS Style Kit Jinja Templatespackage (for layouts, etc.) and
application specific templates from thescar_add_metadata_toolbox.templatesmodule.Styles, components and patterns from theBAS Style Kitshould be used where possible.
Configuration options for Style Kit Jinja Templates are set in thescar_add_metadata_toolbox.configmodule, including
loading local styles and scripts defined inscar_add_metadata_toolbox/static.Application views should inherit from the application layout,app.j2, and usingincludesandmacrosto breakdown and reuse content within views is
strongly encouraged.Editor supportPyCharmMultiple run/debug configurations are included in the project for debugging and testing:Appruns the Flask application and is useful for debug the server role of this application (e.g. CSW requests)App CLIruns the Flask application and is useful for debug the client role of this application (e.g. static site
commands requests) - change theparametersoption to set which command to runTestingAll code in thescar_add_metadata_toolboxpackage must be covered by tests, defined intests/. This project usesPyTestwhich should be ran in a random order usingpytest-random-order.To run tests manually from the command line:$dockercomposerunapp-eFLASK_ENV=testingapppytest--random-orderTo run/debug tests using PyCharm, use the includedApp (Tests)run/debug configuration.Tests are ran automatically inContinuous Integration.Test coveragepytest-covis used to measure test coverage.A.coveragercfile is used to omit code from thescar_add_metadata_toolbox.hazmatmodule.To measure coverage manually:$dockercomposerun-eFLASK_ENV=testingapppytest--cov=scar_add_metadata_toolbox--cov-fail-under=100--cov-report=html.Continuous Integrationwill check coverage automatically and fail if less than 100%.Continuous IntegrationAll commits will trigger a Continuous Integration process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.Review appsTo review changes to functionality, commits made in branches will trigger review apps to be created using GitLab's
CI/CD platform, configured in.gitlab-ci.yml.Review apps run asNomad servicesonly, not asCommand line applications.Containers for review apps are built using thedeployment Docker imagebut tagged asreview:[slug],
where[slug]is a reference to the merge request the review app is related to. Images are hosted in the private BAS
Docker Registry (part ofgitlab.data.bas.ac.uk):docker-registry.data.bas.ac.uk/magic/add-metadata-toolbox/deployLimitationsthe URL for review apps point to the Nomad job via it's UI, rather than the managed application, as the port number
for the application is set dynamically and not stored in the Terraform state for the Nomad jobthe application will currently use the production CSW database, therefore recordsMUST NOTbe changed by review
apps, this is currently unenforced but will be when ServiceDesk ticket #42232 is resolvedDeploymentPython packageA project Python package is built byContinuous Delivery, hosted through the private BAS Repo
Server:bsl-repoa.nerc-bas.ac.uk/magic/v1/projects/scar-add-metadata-toolbox/latest/dist/Docker imageA deployment container image, defined by./support/docker-packaging/Dockerfile, is built byContinuous Deliveryfor releases (Git tags). Images are tagged as/release:[tag], where[tag]is the name of the Git tag a release is related to. Images are hosted in the private BAS Docker Registry (part
ofgitlab.data.bas.ac.uk):docker-registry.data.bas.ac.uk/magic/add-metadata-toolbox/deployNote:All container builds (including those fromReview apps) are also tagged as/build:[commit],
where[commit]is a reference to the Git commit that triggered the image to be built.Docker image expiration policyAn imageexpiration policyis used to
limit the number of non-release container images that are kept. This policy is set within, and enforced automatically
by, GitLab. See theSetup sectionfor how this is configured.Nomad serviceThe deploymentDocker imageis deployed as a service job in the experimentalMAGIC Nomad cluster(internal).BAS IT serviceThe deploymentPython packageis deployed as a WSGI application via BAS IT using an Ansible playbook:/playbooks/magic/add-metadata-toolbox.yml(internal)Variables for this application are set in:/group_vars/magic/add-metadata-toolbox.yml(internal)Environment variables used by this application are set in:/playbooks/magic/add-metadata-toolbox.yml(internal)This application is deployed to a development, staging and production environment. Hosts for each environment are listed
in the relevant Ansible inventory in:/inventory/magic/(internal)Note:The process to run/update this playbook/variables is still under development (see#44(internal) for background). Currently
either needs to be requested through theIT ServiceDesk.Key pathsKey files/directories within this deployed application are:/etc/httpd/sites/10-add-metadata-toolbox.conf: Apache virtual host/var/opt/wsgi/.virtualenvs/add-metadata-toolbox: Python virtual environment/var/www/add-metadata-toolbox/app.py: Application entrypoint script/var/log/httpd/access_log.add_metadata_toolbox: Apache virtual host access log/var/log/httpd/error_log.add_metadata_toolbox: Apache/Application error/log fileSSH accessEnvironmentSSH AccessSudoAccessDevelopmentYesYesfelnneStagingYes (for logs)NofelnneProductionYes (for logs)NofelnneCurrently access to the servers for each environment is bespoke but should be standardised in future, see#100for more information.Flask CLITo use the Flask CLI:$ssh[server]$sudosu
$.[pathtovirtualenvironment]/bin/activate
$exportFLASK_APP=scar_add_metadata_toolbox
$exportFLASK_ENV=production
$flask[command]$deactivate
$exit$exitAPI ServiceThe CSW Catalogues are deployed as a service within the BAS API Load Balancer, backed by the productionBAS IT service.API DocumentationUsage documentation for this API service is held indocs/api/and currentlymanuallypublished using these service paths:s3://bas-api-docs-content-testing/services/data/metadata/add/csw/s3://bas-api-docs-content/services/data/metadata/add/csw/Command line applicationThe deploymentDocker imageis made available as a command line application on the BAS central
workstations using Podman. A wrapper shell script is used to mask thepodman runrun command for ease of use.Continuous DeploymentAll commits will trigger a Continuous Deployment process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.Release procedureFor all releases:create a release branchclose release inCHANGELOG.mdpush changes, merge the release branch intomasterand tag with versioncreate a ServiceDesk request to deploy the new package version (and change/add environment variables if needed)re-deploy API documentation if neededFeedbackThe maintainer of this project is the BAS Mapping and Geographic Information Centre (MAGIC), they can be contacted at:magic@bas.ac.uk.Issue trackingThis project uses issue tracking, see theIssue trackerfor more information.Note:Read & write access to this issue tracker is restricted. Contact the project maintainer to request access.License© UK Research and Innovation (UKRI), 2020 - 2021, British Antarctic Survey.You may use and re-use this software and associated documentation files free of charge in any format or medium, under
the terms of the Open Government Licence v3.0.You may obtain a copy of the Open Government Licence athttp://www.nationalarchives.gov.uk/doc/open-government-licence/