Package: bas-metadata-library
BAS Metadata LibraryPython library for generating metadata and data records.OverviewPurposeThis library is designed to assist in generating metadata and data records, primarily for the discovery of datasets,
services, features and related resources. This project is intended to be used as a dependency, to avoid the need to
duplicate the implementation of complex and verbose metadata and data standards.At a high level, this library allows a configuration object, representing the fields/structure of a standard, to be
encoded into its formal representation set out by that standard (typically using XML). It also allows such a formal
representation to be decoded back into a configuration object, which can be more easily used or manipulated.Supported standardsStandardImplementationLibrary NamespaceIntroduced InISO 19115:2003ISO 19139:2007bas_metadata_library.standards.iso_19115_1_v1#46ISO 19115-2:2009ISO 19139-2:2012bas_metadata_library.standards.iso_19115_2_v1#50IEC 61174:2015IEC 61174:2015bas_metadata_library.standards.iec_pas_61174_0_v1#139IEC PAS 61174:2021IEC PAS 61174:2021bas_metadata_library.standards.iec_pas_61174_1_v1#139Note:In this library, theISO 19115:2003standard is referred to asISO-19115-1(iso_19115_1) for
consistency withISO 19115-2:2009(referred to asISO-19115-2,iso_19115_2). In the future, theISO 19115-1:2014standard will be referred to asISO-19115-3.Supported profilesStandardProfileImplementationLibrary NamespaceIntroduced In-----Note:Support for profiles has been removed to allow underlying standards to be implemented more easily, and to
wait until a stable profile for UK PDC Discovery metadata has been developed and approved.Supported configuration versionsStandardProfileConfiguration VersionStatusNotesISO 19115:2003-v1RetiredReplaced byv2ISO 19115:2003-v2LiveStable versionISO 19115:2003-v3AlphaExperimentalISO 19115-2:2009-v1RetiredReplaced byv2ISO 19115-2:2009-v2LiveStable versionISO 19115-2:2009-v3AlphaExperimentalIEC 61174:2015-v1AlphaExperimentalIEC PAS 61174:2021-v1AlphaExperimentalSupported standards coverageThis library is built around the needs of the British Antarctic Survey and the NERC (UK) Polar Data Centre. This means
only standards, and elements of these standards, used by BAS or the UK PDC are supported. However, additions that would
enable this library to be useful to other organisations and use-case are welcome as contributions providing they do not
add significant complexity or maintenance.StandardCoverageCoverage SummaryISO 19115:2003GoodAll mandatory elements are supported with a good number of commonly used additional elementsISO 19115-2:2009MinimalNo elements from this extension are supported, with the exception of the root elementIEC 61174:2015MinimalAll mandatory elements are supported, plus a limited number of optional route information attributesIEC PAS 61174:2021MinimalAll mandatory elements are supported, plus a limited number of optional route information attributesNote:ISO 19115 extensions (i.e.gmd:metadataExtensionInfoelements) are not supported.Coverage for IEC 61174As required by the IEC 61174 standard, this library supports the following properties within this standard:ElementReferenceObligationroute4.5.2Mandatoryroute.routeInfo.routeAuthor4.5.3Optionalroute.routeInfo.routeName4.5.3Mandatoryroute.routeInfo.routeStatus4.5.3Optionalroute.waypoints4.5.4Mandatoryroute.waypoints.*.waypoint.id4.5.6Mandatoryroute.waypoints.*.waypoint.revision4.5.6Mandatoryroute.waypoints.*.waypoint.position.lat4.5.6Mandatoryroute.waypoints.*.waypoint.position.lon4.5.6Mandatoryroute.waypoints.*.waypoint.position.geometry4.5.6OptionalThis list is exhaustive. No extensions are supported.References in the above table relate to the IEC PAS 61174:2021 standards document:https://webstore.iec.ch/publication/67774.Full citation:IEC 61174:2015, Maritime navigation and radiocommunication equipment and systems – Electronic chart display and
information system (ECDIS) – Operational and performance requirements, methods of testing and required test resultsInstallationThis package can be installed using Pip fromPyPi:$ pip install bas-metadata-libraryThis package depends on native libraries for XML encoding and decoding:libxml2libxsltThis package depends on native binaries for XML validation:xmllintMost Operating Systems include these libraries and packages by default. However, others, particularly minimal OSes
require these packages to be installed separately. Required packages for supported Operating Systems are:Operating SystemRequired PackagesNotesAlpine Linux (Docker)libxslt-dev,libxml2-utils-CentOS 7-Installed by defaultCentOS 7 (Docker)-Installed by defaultUsageEncode an ISO 19115 metadata recordTo generate an ISO 19115 metadata record from a Python record configuration and return it as an XML document:fromdatetimeimportdatefrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3,MetadataRecordminimal_record_config={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration=MetadataRecordConfigV3(**minimal_record_config)record=MetadataRecord(configuration=configuration)document=record.generate_xml_document()# output documentprint(document.decode())See theHTML Entitiessection for guidance on using accents and symbols in descriptions.You will need to use adate_precisionproperty for partial dates. See theDate Precisionsection
for more information.Decode an ISO 19115 metadata recordfrombas_metadata_library.standards.iso_19115_2importMetadataRecordwithopen(f"minimal-record.xml")asrecord_file:record_data=record_file.read()record=MetadataRecord(record=record_data)configuration=record.make_config()minimal_record_config=configuration.config# output configurationprint(minimal_record_config)Upgrade a version 2 ISO 19115 metadata record configuration to version 3The version 3 record configuration object includes an upgrade method. This method accepts a version 2 record and returns
a version 3 object. This method will change the record configuration structure to account for changes introduced in the
version configuration schema.fromdatetimeimportdatefrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV2,MetadataRecordConfigV3minimal_record_config_v2={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration_v2=MetadataRecordConfigV2(**minimal_record_config_v2)configuration_v3=MetadataRecordConfigV3()configuration_v3.upgrade_from_v2_config(v2_config=configuration_v2)Downgrade a version 3 ISO 19115 metadata record configuration to version 2The version 3 record configuration object includes a downgrade method. This method accepts returns a version 2
equivalent of the record configuration.Note: This will result in data loss, in that the V3 configuration allows information that the V2 configuration
does not. This additional information will be lost when downgrading to V2, even if the resulting V2 configuration is
upgraded to V3 again.Information that will be lost when downgrading:any resource constraints with apermissionsproperty - the entire constraint will be lostfromdatetimeimportdatefrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3minimal_record_config_v3={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration_v3=MetadataRecordConfigV3(**minimal_record_config_v3)configuration_v2=configuration_v3.downgrade_to_v2_config()Encode an IEC 61174 route information recordTo encode to a RTZ file:frombas_metadata_library.standards.iec_pas_61174_0_v1importMetadataRecordConfigV1,MetadataRecordminimal_record_config={"route_name":"minimal-test-route","waypoints":[{"id":1001,"revision":0,"position":{"lat":5,"lon":50}},{"id":1002,"revision":0,"position":{"lat":5,"lon":50}},{"id":1003,"revision":0,"position":{"lat":5,"lon":50}},],}configuration=MetadataRecordConfigV1(**minimal_record_config)record=MetadataRecord(configuration=configuration)document=record.generate_xml_document()# output documentprint(document.decode())To encode to a RTZP package:frompathlibimportPathfrombas_metadata_library.standards.iec_pas_61174_0_v1importMetadataRecordConfigV1,MetadataRecordoutput_path=str('/path/to/file.rtzp')minimal_record_config={"route_name":"minimal-test-route","waypoints":[{"id":1001,"revision":0,"position":{"lat":5,"lon":50}},{"id":1002,"revision":0,"position":{"lat":5,"lon":50}},{"id":1003,"revision":0,"position":{"lat":5,"lon":50}},],}configuration=MetadataRecordConfigV1(**minimal_record_config)record=MetadataRecord(configuration=configuration)record.generate_rtzp_archive(file=Path(output_path))Decode an IEC 61174 route information recordTo decode from a RTZ file:frombas_metadata_library.standards.iec_pas_61174_0_v1importMetadataRecordwithopen(f"minimal-record.rtz")asrecord_file:record_data=record_file.read()record=MetadataRecord(record=record_data)configuration=record.make_config()minimal_record_config=configuration.config# output configurationprint(minimal_record_config)To decode from a RTZP package:frompathlibimportPathfrombas_metadata_library.standards.iec_pas_61174_0_v1importMetadataRecordinput_path=str('/path/to/file.rtzp')record=MetadataRecord()record.load_from_rtzp_archive(file=Path(input_path))configuration=record.make_config()minimal_record_config=configuration.config# output configurationprint(minimal_record_config)Loading a record configuration from JSONThe example below is for the ISO 19115 standard but this applies to all standards.Theload()andloads()methods on the configuration class can be used to load a record configuration encoded as a
JSON file or JSON string respectively:frompathlibimportPathfrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3input_path=str('/path/to/file.json')configuration=MetadataRecordConfigV3()configuration.load(file=Path(input_path))Dumping a record configuration to JSONThe example below is for the ISO 19115 standard but this applies to all standards.Thedump()anddumps()methods on the configuration class can be used to dump a record configuration to a JSON
encoded file or string respectively:fromdatetimeimportdatefrompathlibimportPathfrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3output_path=str('/path/to/file.json')minimal_record_config={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration=MetadataRecordConfigV3(**minimal_record_config)configuration.dump(file=Path(output_path))Validating a recordThe example below is for the ISO 19115 standard but this applies to all standards.The formal encoding of a record can be validated against one or more XML schemas relevant to each metadata or data
standard. Records are not validated automatically, and so must be validated explicitly:fromdatetimeimportdatefrombas_metadata_libraryimportRecordValidationErrorfrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3,MetadataRecordminimal_record_config={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration=MetadataRecordConfigV3(**minimal_record_config)record=MetadataRecord(configuration=configuration)try:record.validate()exceptRecordValidationErrorase:print('Record invalid')print(e)Where the contents of the record is invalid, aRecordValidationErrorexception will be raised. Printing this
exception will return validation errors.These errors should not happen, and if they do are considered internal bugs. Please report any in theProject Issue Trackerif you are internal to BAS, or asFeedbackif you are not.See theRecord Schemassection for more information on how validation works.Validating a record configurationThe example below is for the ISO 19115 standard but this applies to all standards.Record configurations will be validated automatically using a JSON Schema relevant to each metadata or data standard.To explicitly validate a record configuration:fromdatetimeimportdatefromjsonschemaimportValidationErrorfrombas_metadata_library.standards.iso_19115_2importMetadataRecordConfigV3minimal_record_config={"hierarchy_level":"dataset","metadata":{"language":"eng","character_set":"utf-8","contacts":[{"organisation":{"name":"UK Polar Data Centre"},"role":["pointOfContact"]}],"date_stamp":date(2018,10,18),},"identification":{"title":{"value":"Test Record"},"dates":{"creation":{"date":date(2018,1,1),"date_precision":"year"}},"abstract":"Test Record for ISO 19115 metadata standard (no profile) with required properties only.","character_set":"utf-8","language":"eng","topics":["environment","climatologyMeteorologyAtmosphere"],"extent":{"geographic":{"bounding_box":{"west_longitude":-45.61521,"east_longitude":-27.04976,"south_latitude":-68.1511,"north_latitude":-54.30761,}}},},}configuration=MetadataRecordConfigV3(**minimal_record_config)try:configuration.validate()exceptValidationErrorase:print('Record configuration invalid')print(e)Where the contents of the record is invalid, aValidationErrorexception will be raised by the underlying JSON Schema library. Printing this exception will return validation errors.See theRecord Configuration Schemassection for more information.HTML entitiesDo not include HTML entities in input to this library, as they will be double escaped byLxml, the
underlying XML processing library used by this project. Instead, literal characters should be used (e.g.>), which
will be escaped as needed automatically. This applies to any unicode character, such as accents (e.g.å) and
symbols (e.g.µ).E.g. If&gt;, the HTML entity for>(greater than), were used as input, it would be escaped again to&amp;gt;which will not be valid output.Date PrecisionWhen using Python for record configurations, date or date times must be structured as dictionaries with adatevalue (which can be a Python date or date time object), and optionaldate_precisionproperty for indicating elements
in the date or date time object should be ignored when encoding records. This property can be set to either:year(month and day are unknown)month(day is unknown)When decoding a record, partial dates or date times will be detected and adate_precisionproperty added
automatically. Unknown elements of a date or date time should, or will, use '1' as a conventional value, which can
effectively be ignored. This is necessary as Python does not allow unknown date elements to be omitted.When using JSON for record configurations, date or date times must be written as strings. Partial dates or date
times can be expressed naturally (e.g.2012-04), without the need for adate_precisionproperty. This library will
automatically convert strings to or from dictionaries, with adate_precisionproperty if needed, when loading from,
or saving to, JSON.Summary table:Date PrecisionPython EncodingJSON EncodingXML EncodingYear{'date': date(year=2012, month=1, day=1), 'date_precision': 'year'}"2012""2012"Month{'date': date(year=2012, month=4, day=1), 'date_precision': 'month'}"2012-04""2012-04"Day{'date': date(year=2012, month=4, day=14)}"2012-04-14""2012-04-14"Hour{'date': datetime(year=2012, month=4, day=14, hour=6)}"2012-04-14T06""2012-04-14T06"Minute{'date': datetime(year=2012, month=4, day=14, hour=6, minute=30)}"2012-04-14T06:30""2012-04-14T06:30"Second{'date': datetime(year=2012, month=4, day=14, hour=6, minute=30, second=42)}"2012-04-14T06:30:42""2012-04-14T06:30:42"Note:For date times, the UTC timezone is assumed unless set.ISO 19115 - linkages between transfer options and formatsIn ISO 19115, there is no formal mechanism to associate file distribution formats and transfer options. As this library
seeks to be fully reversible between a configuration object and formal XML encoding, associations between these elements
would be lost when records are encoded as XML. These associations are used to produce download tables such as [1].InRecord Configurations, these associations are encoded using a 'distribution option'
concept. In formal XML records, these associations are encoded usingxsd:IDattributes ingmd:MD_Formatandgmd:DigitalTransferOptionselements, with values that allow these associations to be reconstructed when decoding XML.Note:Do not modify automatically assigned IDs, as this will break this functionality.See theAutomatic transfer option / format IDssection for more
information.[1]FormatSizeDownload LinkCSV68 kBLinkGeoPackage1.2 MBLinkImplementationThis library is implemented in Python and consists of a set of classes used to generate XML metadata and data records
from a configuration object, or to generate a configuration object from an XML record.Eachsupported Standardis implemented as a module underbas_metadata_library.standards. EachSupported Profileis implemented as modules under their respective standard.Base classesFor each standard and profile, instances of these base classes are defined:NamespacesMetadataRecordMetadataRecordConfigThenamespacesclass is a set of mappings between XML namespaces, their shorthand aliases and their definitions XSDs.TheMetadataRecordclass represents a metadata record and defines the RootElement. This
class provides methods to generate an XML document for example.TheMetadataRecordConfigclass represents theConfigurationused to define values within aMetadataRecord, either for new records, or derived from existing records. This class provides methods to validate the
configuration used in a record for example.Record element classesEach supported element, in eachsupported standard, inherit and use theMetadataRecordElementclass to:encode configuration values into an XML fragment of at least one elementdecode an XML fragment into one or more configuration valuesSpecifically, at least two methods are implemented:make_element()which builds an XML element using values from a configuration objectmake_config()which uses typically XPath expressions to build a configuration object from XMLThese methods may be simple (if encoding or decoding a simple free text value for example), or quite complex, using
sub-elements (which themselves may contain sub-elements as needed).Record schemasAllowed elements, attributes and values for eachsupported StandardandSupported Profileare defined using one or moreXML Schemas.
These schemas define any required entities, and any entities with enumerated values. Schemas are usually published by
standards organisations to facilitate record validation.For performance reasons, and to ensure required schemas are not unavailable (due to remote locations being reorganised,
or during server maintenance etc.), these schema files are stored within this package. Schemas are stored as XML Schema
Definition (XSD) files in thebas_metadata_library.schemas.xsdmodule, and loaded as resource files for use in record
validation.Note:To support local validation, imported or included schema locations in local versions of XML schemas, have
been modified. These changes do not materially change the contents of any schema.Note:In some cases, material changeshavebeen made to local versions of schemas, in order to workaround
specific issues. These changes will be documented and explained, to allow users to understand the effect they will
have, and why they have been made.Altered Metadata Schema - Geographic Metadata (GMD)The ISOGeographic Metadata (GMD)schema (used directly for the ISO 19115-0 standard, and indirectly in the ISO
19115-2 standard) has been modified to:include the ISOGeographic Metadata XML (GMX)schema:in order to allow Anchor elements to substitute primitive/simple values (such as character strings and integers),as defined in the ISO 19139:2007 and ISO 19139:2012 standardsConfiguration classesThe configuration of each metadata record is held in a Python dictionary, within aMetadataRecordConfigclass. This
class includes methods to validate its configuration against a relevantConfiguration Schema.Configuration classes are defined at the root of each standard or profile, alongside its rootMetadata Elementand XML namespaces.A configuration class will exist for each supported configuration schema with methods to convert from one version to
another.Configuration schemasAllowed properties and values for record configurations for eachsupported StandardandSupported Profileare defined using aJSON Schema. These schemas
define any required properties, and any properties with enumerated values.Configuration schemas are stored as JSON files in thebas_metadata_library.schemasmodule, and loaded as
resource files from within this package to validate record configurations. Schemas are also made available externally
through the BAS Metadata Standards website,metadata-standards.data.bas.ac.uk, to allow:other applications to ensure their output will be compatible with this library - where they can't, or don't want to,
use this library directlyschema inheritance/extension - for standards that inherit from other standards (such as extensions or profiles)Configuration schemas are versioned (e.g.v1,v2) to allow for backwards incompatible changes to be made.
Upgrade/Downgrade methods will be provided for a limited time to assist migrating record configurations between schema
versions.Source and distribution schemasStandards and profiles usually inherit from other standards and profiles. In order to prevent this creating huge
duplication within configuration schemas, inheritance is used to incorporate a base schema and extend it as needed. For
example, the ISO 19115-2 standard extends, and therefore incorporates the configuration schema for, ISO 19115-1.JSON Schema references and identifier properties are used to implement this, using URIs within the BAS Metadata
Standards website. Unfortunately, this creates a problem when developing these schemas, as if Schema B relies on Schema
A, using its published identifier as a reference, the published instance of the schema will be used (i.e. the remote
schema will be downloaded when Schema B is validated). If Schema A is being developed, and is not ready to be
republished, there is a difference between the local and remote schemas used, creating unreliable tests for example.To avoid this problem, a set ofsourceschemas are used which use references to avoid duplication, from which a set
ofdistributionschemas are generated. These distribution schemas inline any references contained in their source
counterpart. These distribution schemas are therefore self-contained and can be updated locally without any
dependencies on remote sources. Distribution schemas are used byConfiguration Classesand
published to the BAS Metadata Standards website, they are located in thebas_metadata_library.schemas.distmodule.When editing configuration schemas, you should edit the source schemas, located in thebas_metadata_library.schemas.srcmodule, thengenerate distribution schemas.JSON Schema's can be developed usingjsonschemavalidator.net.Adding a new standardTo add a new standard:create a new module underbas_metadata_library.standards, e.g.bas_metadata_library.standards.foo_v1/__init__.pyin this module, overload theNamespaces,MetadataRecordConfigandMetadataRecordclasses as neededversion theMetadataRecordConfigclass, e.g.MetadataRecordConfigV1create a suitable metadata configuration JSON schema inbas_metadata_library.schemas.src,
e.g.bas_metadata_library.schemas.src.foo_v1.jsonupdate thegenerate_schemasmethod inapp.pyto generate distribution schemasadd a script line to thepublish-schemas-stageandpublish-schemas-prodjobs in.gitlab-ci.yml, to publish
the distribution schema within the BAS Metadata Standards websitedefine a series of test configurations (e.g. minimal, typical and complete) for generating test records intests/resources/configs/e.g.tests/resources/configs/foo_v1_standard.pyadd a route inapp.pyfor generating test records for the new standardupdate thecapture_test_recordsmethod inapp.pyto generate and save test recordsadd relevanttestswith methods to test each metadata element class and test recordsAdding a new element to an existing standardNote:These instructions are specific to the ISO 19115 metadata standards family.amend configuration schema:new or changed properties should be added to the configuration for the relevant standard (e.g. ISO 19115-1)typically, this involves adding new elements to thedefinitionsproperty and referencing these in the relevant
parent element (e.g. to theidentificationproperty)generate distribution schemasamend test configs:new or changed properties should be made to the relevant test record configurations intests/resources/configs/there are different levels of configuration, from minimal to complete, which should, where possible, build on
each other (e.g. the complete record should include all the properties and values of the minimal record)theminimumconfiguration should not be changed, as all mandatory elements are already implementedthebase_simpleconfiguration should contain elements used most of the time, that use free-text valuesthebase_complexconfiguration should contain elements used most of the time, that use URL or other
identifier valuesthecompleteconfiguration should contain examples of all supported elements, providing this still produces a
valid record, in order to ensure high test coveragewhere possible, configurations should be internally consistent, but this can be ignored if neededvalues used for identifiers and other external references should use the correct form/structure but do not need
to exist or relate to the resource described by each configuration (i.e. DOIs should be valid URLs but could be
a DOI for another resource for example)add relevantelement class:new or changed elements should be added to the configuration for the relevant package for each standardfor the ISO 19115 family of standards, element classes should be added to theiso_19115_commonpackagethe exact module to use within this package will depend on the nature of the element being added, but in general,
elements should be added to the module of their parent element (e.g.data_identification.pyfor elements
under theidentificationrecord configuration property), elements used across a range of elements should be
added to thecommon_elements.pymoduleremember to include references to new element class in the parent element class (in both themake_elementandmake_configmethods)capture test recordsinitially this acts as a good way to check new or changed element classes encode configuration properties
correctlycheck the git status of these test records to check existing records have changed how you expect (and haven't
changed things you didn't intend to for example)capture test JSON configurationscheck the git status of these test configs to check they are encoded correctly from Python (i.e. dates)add tests:new test cases should be added, or existing test cases updated, in the relevant module withintests/bas_metadata_library/for the ISO 19115 family of standards, this should betest_standard_iso_19115_1.py, unless the element is only
part of the ISO 19115-2 standardproviding there are enough test configurations to test all the ways a new element can be used (e.g. with a simple
text string or anchor element for example), adding a test case for each element is typically enough to ensure
sufficient test coveragewhere this isn't the case, it's suggested to add one or more 'edge case' test cases to test remaining code paths
explicitlychecktest coverage:for missing coverage, consider adding edge case test cases where applicablecoverage exemptions should be avoided wherever feasible and all exemptions must be discussed before they are addedwhere exceptions are added, they should be documented as an issue with information on how they will be addressed
in the longer termupdateREADME.mdexamples if common element:this is probably best done before releasing a new versionupdateCHANGELOG.mdif needed, add name toauthorsproperty inpyproject.tomlAdding a new config version for an existing standard [WIP]Note:This is typically only needed if breaking changes need to be made to the schema for a configuration, as the
work involved is quite involved.Note:This section is a work in progress whilst developing the ISO 19115 v3 configuration in#182.Note:In these instructions,v1refers to the current/previous configuration version.v2refers to the new
configuration version.First create a new configuration version that is identical to the current/previous version, but that sets up the
schema, objects, methods, tests and documentation needed for the new configuration, and to convert between the old
and new configurations.create an issue summarising, and referencing specific issues for, changes to be made in the new schema versioncopy the current/previous metadata configuration JSON schema frombas_metadata_library.schemas.srce.g.bas_metadata_library.schemas.src.foo_v1.jsontobas_metadata_library.schemas.src.foo_v2.jsonchange the version in:the$idpropertythetitlepropertythedescriptionpropertyduplicate the configuration classes for the standard inbas_metadata_library.standardsi.e. inbas_metadata_library.standards.foo_v1/__init__.py, copy:MetadataRecordConfigV1toMetadataRecordConfigV2in the new configuration class, addupgrade_to_v1_config()anddowngrade_to_v2_config()methodstheupgrade_from_v2_config()method should accept a current/previous configuration classthedowngrade_to_v1_config()method should return a current/previous configuration classchange the signature of theMetadataRecordclass to use the new configuration classchange themake_config()method of theMetadataRecordclass to return the new configuration classupdate thegenerate_schemas()method inapp.pyto generate distribution schemas for the new schema versionGenerate configuration schemasadd a script line to thepublish-schemas-stageandpublish-schemas-prodjobs in.gitlab-ci.yml, to publish
the distribution schema for the new schema version within the BAS Metadata Standards websitedefine a series of test configurations (e.g. minimal, typical and complete) for generating test records intests/resources/configs/e.g.tests/resources/configs/foo_v1_standard.pynote that the version in these file names is for the version of the standard, not the configurationnew config objects will be made within this file that relate to the new configuration versioninitially these new config objects can inherit from test configurations for the current/previous versionupdate thegenerate_json_test_configs()method inapp.pyto generate JSON versions of each test configurationCapture test JSON record configurationsupdate the route for the standard inapp.py(e.g.standard_foo_v1) to:upgrade configs for the old/current version of the standard (as the old/current MetadataRecordConfig class will
now be incompatible with the updated MetadataRecord class)include configs for the new config version of the standardupdate thecapture_test_records()method inapp.pyto capture test records for the new test configurationsCapture test XML recordsadd test cases for the newMetadataRecordConfigclass in the relevant module intests.bas_metadata_library:test_invalid_configuration_v2test_configuration_v2_from_json_filetest_configuration_v2_from_json_stringtest_configuration_v2_to_json_filetest_configuration_v2_to_json_stringtest_configuration_v2_json_round_triptest_parse_existing_record_v2test_lossless_conversion_v2change all test cases to target record configurations for the new versionupdate thetest_record_schema_validation_validandtest_record_schema_validation_validtest cases, which test
the XML/XSD schema for the standard, not the configuration JSON schemaupdate the existingtest_lossless_conversion_v1test case to upgrade v1 configurations to v2, as theMetadataRecordclass will no longer be compatible with theMetadataRecordConfigV1classupdate theSupported configuration versionssection of the READMEadd the new schema version, with a status of 'alpha'update the encode/decode subsections in theUsagesection of the README to use the new RecordConfig classif the lead standard (ISO 19115) is being updated also update theseUsagesubsections:Loading a record configuration from JSONDumping a record configuration to JSONValidating a recordValidating a record configurationadd a subsection to theUsagesection of the README explaining how to upgrade and downgrade a
configuration between the old and new versionsUpdate the change log to reference the creation of the new schema version, referencing the summary issueSecond, iteratively introduce changes to the new configuration, adding logic to convert between the old and new
configurations as needed. This logic will likely be messy and may target specific known use-cases. This is acceptable on
the basis these methods will be relatively short lived.as changes are made, add notes and caveats to the upgrade/downgrade methods in code, and summarise any
significant points in theUsageinstructions as needed (e.g. that the process is lossy)ISO 19115 - Automatic transfer option / format IDsID attributes are automatically added togmd:MD_Formatandgmd:MD_DigitalTransferOptionselements in order to
reconstruct related formats and transfer options (see theLinking transfer options and formatssection for more
information).When a record is encoded, ID values are generated by hashing a JSON encoded string of the distribution object. This
ID is used as a shared base between the format and transfer option, with-fmtappended for the format and-tfofor the transfer option.When a record is decoded, ID values are extracted (stripping the-fmt/-tfosuffixes) to index and then match up
format and transfer options back into distribution options. Any format and transfer options without an ID value, or
without a corresponding match, are added as partial distribution options.As a worked example for encoding a (simplified) distribution object such as:do={'format':'csv','transfer_option':{'size':'40','url':'https://example.com/foo.csv'}}Becomes:'{"format":"csv","transfer_option":{"size":40,"url":"https://example.com/foo.csv"}}'When encoded as a JSON encoded string, which when hashed becomes:16b7b5df78a664b15d69feda7ccc7caed501f341The ID value added to thegmd:MD_Formatelement would be:<gmd:MD_Formatid="bml-16b7b5df78a664b15d69feda7ccc7caed501f341-fmt">And for thegmd:MD_DigitalTransferOptionselement:<gmd:MD_DigitalTransferOptionsid="bml-16b7b5df78a664b15d69feda7ccc7caed501f341-tfo">Thebml-prefix is added to ensure all IDs begin with a letter (as required by XML), and to allow IDs generated by
this library to be detected. The-fmt/-tfoprefixes are used to allow the same ID value to uniquely identify two
elements uniquely.SetupTerraformTerraform is used to provision resources required to operate this application in staging and production environments.These resources allowConfiguration schemasfor each standard to be accessed externally.Access to theBAS AWS accountis needed to provision these resources.Note:This provisioning should have already been performed (and applies globally). If changes are made to this
provisioning it only needs to be applied once.# start terraform inside a docker container$cdprovisioning/terraform
$dockercomposerunterraform# setup terraform$terraforminit# apply changes$terraformvalidate
$terraformfmt
$terraformapply# exit container$exit$dockercomposedownTerraform remote stateState information for this project is stored remotely using aBackend.Specifically theAWS S3backend as part of theBAS Terraform Remote Stateproject.Remote state storage will be automatically initialised when runningterraform init. Any changes to remote state will
be automatically saved to the remote backend, there is no need to push or pull changes.Remote state authenticationPermission to read and/or write remote state information for this project is restricted to authorised users. Contact
theBAS Web & Applications Teamto request access.See theBAS Terraform Remote Stateproject for how these
permissions to remote state are enforced.DevelopmentThis API is developed as a Python library. A bundled Flask application is used to simulate its usage, act as
framework for running tests etc., and provide utility methods for generating schemas etc.Development environmentGit andPoetryare required to set up a local development environment of this application.Note:If you usePyenv, this project sets a local Python version for consistency.If you have access to theBAS GitLab instance:# clone from the BAS GitLab instance if possible$gitclonehttps://gitlab.data.bas.ac.uk/uk-pdc/metadata-infrastructure/metadata-library.git# alternatively, clone from the GitHub mirror$gitclonehttps://github.com/antarctica/metadata-library.git# setup virtual environment$cdmetadata-library
$poetryinstallCode StylePEP-8 style and formatting guidelines must be used for this project, except the 80 character line limit.Blackis used for formatting, configured inpyproject.tomland enforced as part ofPython code linting.Black can be integrated with a range of editors, such asPyCharm, to apply formatting
automatically when saving files.To apply formatting manually:$poetryrunblacksrc/tests/Code Linting (Python)Flake8and various extensions are used to lint Python files in thebas_metadata_librarymodule. Specific checks, and any configuration options, are documented in the./.flake8config file.To check files manually:$poetryrunflake8src/Checks are run automatically inContinuous Integration.Code Linting (JSON)JSON files (specifically JSON Schemas used forRecord Configurations) must be valid JSON
documents. Minimal linting is used to enforce this as part ofContinuous Integration.To check files manually:$forfilein$(find./src/bas_metadata_library/schemas/src-name"*.json");doecho${file};poetryrunpython-mjson.tool<${file}1>/dev/null;doneDependenciesPython dependencies for this project are managed withPoetryinpyproject.toml.Non-code files, such as static files, can also be included in thePython packageusing theincludekey inpyproject.toml.Adding new dependenciesTo add a new (development) dependency:$poetryadd[dependency](--dev)Then update the Docker image used for CI/CD builds and push to the BAS Docker Registry (which is provided by GitLab):$dockerbuild-fgitlab-ci.Dockerfile-tdocker-registry.data.bas.ac.uk/uk-pdc/metadata-infrastructure/metadata-library:latest.
$dockerpushdocker-registry.data.bas.ac.uk/uk-pdc/metadata-infrastructure/metadata-library:latestUpdating dependencies$poetryupdateSee the instructions above to update the Docker image used in CI/CD.Dependency vulnerability checksTheSafetypackage is used to check dependencies against known vulnerabilities.IMPORTANT!As with all security tools, Safety is an aid for spotting common mistakes, not a guarantee of secure
code. In particular this is using the free vulnerability database, which is updated less frequently than paid options.This is a good tool for spotting low-hanging fruit in terms of vulnerabilities. It isn't a substitute for proper
vetting of dependencies, or a proper audit of potential issues by security professionals. If in any doubt you MUST seek
proper advice.Checks are run automatically inContinuous Integration.To check locally:$poetryexport--without-hashes-frequirements.txt|poetryrunsafetycheck--full-report--stdinjsonschemapackageThejsonschemadependency is locked to version 3.2.0 because version 4.0 > dropped Python 3.6 support. This
library cannot require newer Python versions to ensure it can be used in projects that run on BAS IT infrastructure.lxmlpackageThelxmldependency takes a long time to install/update inside Alpine container images because it needs to be built
from source. This is because Alpine Linux, used by the official Python Docker base images, is not supported by the
Pythonmanylinuxsystem, and therefore cannot use pre-built, binary, wheels.Static security scanningTo ensure the security of this API, source code is checked againstBanditand enforced as part ofPython code linting.Warning:Bandit is a static analysis tool and can't check for issues that are only be detectable when running the
application. As with all security tools, Bandit is an aid for spotting common mistakes, not a guarantee of secure code.Checks are run automatically inContinuous Integration.lxmlpackage (bandit)Bandit identifies the use oflxmlclasses and methods as a security issue, specifically:Element to parse untrusted XML data is known to be vulnerable to XML attacksThe recommendation is to use asafeimplementation of an XML processor (defusedxml) that can avoid entity bombs and
other XML processing attacks. However,defusedxmldoes not offer all of the methods we need and there does not appear
to be such another processor that does provide them.The main vulnerability this security issue relates to is processing user input that can't be trusted. This isn't really
applicable to this library directly, but rather to where it's used in implementing projects. I.e. if this library is
used in a service that accepts user input, an assessment must be made whether the input needs to be sanitised.Within this library itself, the only input that is processed is test records, all of which are assumed to be safe to
process.Generating configuration schemasTo generatedistribution schemas from source schemas, a custom Flask CLI command,generate-schemasis available. Thejsonreflibrary is used to resolve
any references in source schemas and write the output as distribution schemas, replacing any existing output.$poetryrunflaskgenerate-schemasTo configure this command, (e.g. to add a new schema for a new standard/profile), adjust theschemaslist in thegenerate_schemasmethod inapp.py. This list should contain dictionaries with keys for the common name of the
schema (based on the common file name of the schema JSON file), and whether the source schema should be resolved or
simply copied. This should be true by default, and is only relevant to schemas that do not contain any references, as
this will cause an error if resolved.TestingAll code in thebas_metadata_librarymodule must be covered by tests, defined intests/. This project usesPyTestwhich should be run in a random order usingpytest-random-order.Tests are written to create metadata records based on a series of configurations defined intests/resources/configs/.
These define 'minimal' to 'complete' test records, intended to test different ways a standard can be used, both for
individual elements and whole records. These tests are designed to ensure that records are generally well-formed and
that where config options are used the corresponding elements in the metadata record are generated.As this library does not seek to support all possible elements and variations within each standard, these tests are
similarly not exhaustive, nor are they a substitute for formal metadata validation.Test methods are used to test individual elements are formed correctly. Comparisons against static records are used to
test the structure of whole records.To run tests manually from the command line:$poetryrunpytest--random-orderTo run tests manually using PyCharm, use the includedApp (Tests)run/debug configuration.Tests are run automatically inContinuous Integration.Capturing test recordsTo capture test records, which verify complete records are assembled correctly, a custom Flask CLI command,capture-test-recordsis available. This command will update pre-existing records intests/resources/records, with
differences captured in version control to aid in manual review to ensure changes are expected/correct.$poetryrunflaskcapture-test-recordsCapturing test configurations as JSONTo capture test configurations as JSON, which verify the dump/load methods of configuration classes work encode and
decode information correctly, a custom Flask CLI command,capture-json-test-configsis available. This will dump all
test configurations for each standard to set of JSON files intests/resources/configs/.$poetryrunflaskcapture-json-test-configsThese files MUST then be manually verified to ensure they have encoded each configuration correctly. Once they have,
they can be used in tests to automatically verify this remains the case.It is intended that this command will update pre-existing configurations, with differences captured in version control
to aid in manual review to ensure they are correct.Test coveragepytest-covis used to measure test coverage.To measure coverage manually:$poetryrunpytest--random-order--cov=bas_metadata_library--cov-fail-under=100--cov-report=html.Continuous Integrationwill check coverage automatically and fail if less than 100%.Continuous IntegrationAll commits will trigger a Continuous Integration process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.DeploymentPython packageThis project is distributed as a Python package, hosted inPyPi.Source and binary packages are built and published automatically usingPoetryinContinuous Deployment.Note:Except for tagged releases, Python packages built in CD will use0.0.0as a version to indicate they are
not formal releases.Continuous DeploymentA Continuous Deployment process using GitLab's CI/CD platform is configured in.gitlab-ci.yml.Release procedureFor all releases, create a release issue by creating a newissue, with the 'release' issue template,
and follow its instructions.FeedbackThe maintainer of this project is the BAS Web & Applications Team, they can be contacted at:servicedesk@bas.ac.uk.Issue trackingThis project uses issue tracking, see theIssue trackerfor more
information.Note:Read & write access to this issue tracker is restricted. Contact the project maintainer to request access.LicenseCopyright (c) 2019-2022 UK Research and Innovation (UKRI), British Antarctic Survey.Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.