Package: bas-air-unit-network-dataset
BAS Air Unit Network DatasetManagement of the network of routes and waypoints used by the British Antarctic Survey (BAS) Air Unit.Including a utility to process routes and waypoints for use in handheld and aircraft GPS devices used by the Air Unit.OverviewPurposeTo support the BAS Air Unit manage their network of routes and waypoints such that:information is internally consistent, through defined structures and constraintsinformation is interoperable between different systems, through the use of open/standard formatsinformation is well described and sharable with other teams, through distribution as datasets (through theOps Data Store 🛡for example)Note:This project is focused on needs within the British Antarctic Survey. It has been open-sourced in case it's
of use to others with similar or related needs. Some resources, indicated with a 🛡 symbol, are not accessible publicly.BackgroundThis project was developed in response to discussions and requests with the BAS Air Unit to review and simplify the
process they used to manage their network of waypoints, and to ensure its future sustainability.BAS staff can read more about this background in thisGitLab issue 🛡.StatusThis project is a mature alpha. This means:all, or parts, of this project:may, but should not, stop working (due to regressions or instability)may not work correctly, or as expectedly (including destructively)may change at any time (in terms of implementation or functionality)documentation may be missing or incorrectsupport for this project is provided on a best efforts / 'as is' basisoutputs from this project should not be relied upon for operation use without thorough scrutinyLimitationsThis service has a number of limitations, including:the Air Unit Network utility does not support multiple, or additional, networksthe Air Unit Network utility does not require route names to follow the required naming conventionthe Air Unit Network utility does not require waypoint identifiers to be unique across all waypointsthe Air Unit Network utility does not require waypoint comments to follow the required GPX comment structurethe Air Unit Network utility does not require waypoints within imported routes to be listed as standalone waypointscomments for waypoints use an overly complex structure to support an ad-hoc serialisation format within GPX filesUnicode characters (such as emoji) are unsupported in route/waypoint names, comments, etc.CSV outputs are not designed for printing (i.e. column formatting and page breaks)Some or all of these limitations may be addressed in future improvements to this project. See the projectissue tracker 🛡for details.UsageLoading features from GPX filesIf using a GPX file to load waypoints and routes into a network, for waypoints these requirements must be met in
addition to the constraints from theInformation Model:the GPX comment field should consist of 5 elements, in the order below, separated with a vertical bar (|):name: a full, or formal name for the waypoint (maximum 17 characters)co-located with: name of a related depot, instrument and/or other feature - useN/Aif not relevantlast accessed at: date waypoint was last accessed in the formYYYY-MM-DD- useN/Aif unvisitedlast accessed by: pilot that that last accessed waypoint - useN/Aif unvisitedcomment: any other information - useN/Aif not relevantFor example (a co-located, previously visited, waypoint with a full name and additional information):identifier:ALPHAcomment:Alpha 001 | Dog | 2014-12-24 | CW | Bring treats.For example (a standalone, unvisited, waypoint with no full/formal name or additional information):identifier:BRAVOcomment:N/A | N/A | N/A | N/A | N/ANote:Only the 'name' in a comment will be included in FPL waypoints.Creating outputsSee thetests/create_outputs.pyfor an example of converting a set of input waypoints and
routes into output formats, creating anOutput Directory, using an instance of theMainAirUnitNetworkclass.TheNetworkclass (on which theMainAirUnitNetworkclass is based) includes a built-in method for loading features
from a GPX file (used in the example above). To load data from other data sources, constructWaypointandRoutefeatures directly and add to the Network class.Output directoryWhen using theMainAirUnitNetworkclass from this project to process waypoints and routes, an output directory
similar to the example below will be created. This directory should be held in a suitable location where all relevant
users can access it.A typical/example output directory:/path/to/output/directory
├── CSV
│   ├── 00_WAYPOINTS_DDM_2023_12_03.csv
│   └── 00_WAYPOINTS_DD_2023_12_03.csv
├── FPL
│   ├── 00_NETWORK_2023_12_03.fpl
│   ├── 01_BRAVO_TO_ALPHA.fpl
│   ├── 02_BRAVO_TO_BRAVO.fpl
│   └── 03_BRAVO_TO_LIMA.fpl
└── GPX
    └── 00_NETWORK_2023_12_03.gpxAccess controlThe Air Unit Network utility does not include access control. If needed, access controls should be applied to the
output directory, as is the case for theOps Data Store 🛡for
example.ImplementationThis project consists of:a description and schema for the main BAS Air Unit travel network (routes and waypoints)a Python library to:import waypoints and routes from a GPX file, or other data sourceexport waypoints and routes into a range of output formats (currently CSV, GPX and Garmin FPL)Information modelThe BAS Air Unit Network information model consists of two entities, forming two, related, datasets:Waypoints: Features representing landing sites used by the Air Unit, usually co-located with a BAS Operations
depot, field camp or a science/monitoring instrumentRoutes: Features representing formally defined, frequently travelled, paths between two or more Waypoints, as
opposed to ad-hoc pathsFor example:Waypoints: Fossil BluffRoutes: Rothera to Fossil BluffThere is a many-to-many relationship between Waypoints and Routes. I.e. a Waypoint can be part of many Routes, and
Routes can contain many Waypoints.Note:This information model is abstract and requires implementing. See theData modelsection for
the current implementation.Waypoints (information model)PropertyNameTypeOccurrenceLengthDescriptionExampleidIDString11 - ..Unique identifier'01G7MY680N332AW9H9HR9SG15T'identifierIdentifierString11 - 6Unique reference'ALPHA'geometryGeometryGeometry (2D Point, EPSG:4326)1-Position or location as a single coordinate'SRID=4326;Point(-75.014648 -69.915214)'nameNameString0-11 - 17Full or formal name'Alpha 001'colocated_withCo-located WithString0-11 - ..Features (from other domains) associated with the waypoint'Depot: Foo'last_accessed_atLast Accessed AtDate0-11 - ..When the Waypoint was last accessed or visited'2014-12-24'last_accessed_byLast Accessed ByString0-11 - ..Who last accessed or visited the Waypoint'Conwat'commentCommentString0-11 - ..Freetext description or comments'Alpha 001 is on a high ridge ...'ID (Waypoint)IDs:MUST be uniqueMUST NOT be based on any information contained within the WaypointMAY use any format/scheme:the same scheme SHOULD be used for all IDsnon-sequential schemes are recommendedNote:This ID can be used to refer to each Waypoint in other systems (i.e. as a foreign identifier).Identifiers (Waypoint)Identifiers:MUST be between 1 and 6 uppercase alphanumeric characters without spaces (A-Z, 0-9)MUST be unique across all WaypointsGeometry (Waypoint)Geometries:MUST be expressed in decimal degrees using the EPSG:4326 projectionMUST consist of a longitude (X) and latitude (Y) dimension (2D point)Name (Waypoint)If specified:MUST be between 1 and 17 uppercase alphanumeric or space characters (A-Z, 0-9, ' ')Co-located with (Waypoint)No special comments.Last accessed at (Waypoint)If specified:MUST be expressed as anISO 8601-1:2019date instantLast accessed by (Waypoint)If specified:MUST unambiguously reference an individualMAY use any scheme:the same scheme SHOULD be used for all WaypointsComment (Waypoint)No special comments.Routes (information model)PropertyNameTypeOccurrenceLengthDescriptionExampleidIDString11 - ..Unique identifier'01G7MZB9X0R8S7RTNYAMAQKHE4'nameNameString11 - ..Name or reference'01_ALPHA_TO_BRAVO'waypointsWaypointsList of Waypoint entities2-n-Sequence of Waypoints-ID (Route)IDs:MUST be uniqueMUST NOT be based on any information contained within the RouteMAY use any format/scheme:the same scheme SHOULD be used for all IDsnon-sequential schemes are recommendedNote:This ID can be used to refer to each Route in other systems (i.e. as a foreign identifier).Name (Route)Names:MUST use the format{Sequence}_{First Waypoint Identifier}_TO_{Last Waypoint Identifier}, where{Sequence}is
a zero padded, two character, auto-incrementing prefix (e.g. '01', '02', ..., '99').Waypoints (Route)Waypoints in Routes:MUST be a subset of the set of Waypointsi.e. waypoints in routes MUST be drawn from a common set, rather than defined ad-hoc or inline within a RouteMUST be expressed as a sequence:i.e. a list in a specific order from a start to finish via any number of other placesMAY be included multiple timesi.e. the start and end can be the same Waypoint, or a route may pass through the same waypoint multiple timesData modelFor use within the Python library included in this project, and as a reference to implementors for storing entities, a
data model implementing theInformation modelis available. For the later use-case, this data model assumes the
use of a relational model, specifically for SQLite (as an OGC GeoPackage) and PostgreSQL.This data model uses three entities:Waypoint: Point features with attributesRoute: Features to contextualise a set of Waypoints, with attributes (such as route name)RouteWaypoint: join between a Waypoint and a Route, with contextual attributes (such as sequence within route)Note:This data model does not describe how entities are encoded in specificOutput Formats.FIDsFeature Identifiers (FIDs) are created automatically for features without one. FIDs are unique auto-incrementing
integers, suitable for use as primary keys within relational database.FIDs SHOULD be considered an implementation detail, and SHOULD be ignored in favour of ID properties (i.e. 'ID' rather
than 'FID') outside the specific technology being used.Consequently, FIDs SHOULD NOT be exposed to end users and their values or structure MUST NOT be relied upon.ULIDsUniversally Unique Lexicographically Sortable Identifier (ULID)s are the scheme used
for identifiers (IDs).These IDs MAY be exposed to end users.Waypoints (data model)Python class:Waypoint(single waypoint)WaypointCollection(waypoints set)GeoPackage layer:waypointsPropertyNameData TypeNullableUniqueMax LengthNotesfidFeature IDIntegerNoYes-Internal to database, primary key, auto-incrementingidIDULID (String)NoYes--identiferIdentifierStringNoYes6-geometryGeometry2D PointNoNo--nameNameStringYesNo17-colocated_withCo-located WithStringYesNo--last_accessed_atLast Accessed AtDateYesNo--last_accessed_byLast Accessed ByStringYesNo--commentCommentStringYesNo--Routes (data model)Python class:Route(single route)RouteCollection(routes set)GeoPackage layer:routesPropertyNameData TypeNullableUniqueMax LengthNotesfidFeature IDIntegerNoYes-Internal to database, primary key, auto-incrementingidIDULID (String)NoYes--nameNameStringNoYes--Route Waypoints (data model)Python class:RouteWaypoint(single waypoint in route)GeoPackage layer:route_waypointsPropertyNameData TypeNullableUniqueMax LengthNotesfidFeature IDIntegerNoYes-Internal to database, primary key, auto-incrementingroute_idRoute IDULID (String)NoYes-Foreign key to Route entitywaypoint_idWaypoint IDULID (String)NoYes-Foreign key to Waypoint entitysequenceSequenceIntegerNoYes (within Route)-Position of waypoint within a route, value must be unique within each routeNote:Though theroute_idandwaypoint_idcolumns are effectively foreign keys, though they are not configured
as such within the GeoPackage.Test networkA network consisting of 12 waypoints and 3 routes is used to:test various edge casesprovide consistency for repeatable testingprevent needing to use real data that might be sensitiveWARNING!This test network is entirely fictitious. It MUST NOT be used for any real navigation.The canonical test network is stored intests/resources/test-network/test-network.jsonand is versioned using a date
in themeta.versionproperty. A QGIS project is also provided to visualise the test network and ensure derived
outputs match expected test data.This dataset does not follow any particular standard or output format as it's intended to be a superset of other
formats and support properties that may not be part of any standard. Derived versions of the network in some standard
formats are also available (from the same directory) for testing data loading, etc.Updating test networkIf updating the test network, ensure to:update the version attribute in the test network to the current daterecreate derived versions of the network as needed (for example the GPX derived output) [1]use the network utility to generate sample exports [2]manually verify the QGIS project for visualising the network is correct and update/fix as needed[1]$ poetry run python tests/create_derived_test_outputs.py[2]$ poetry run python tests/create_outputs.pyAfter running, ensure all dates in files are updated to values set intests/compare_outputs.py.Output formatsSupported formatsFormat use-cases:FormatUse CaseCSVHuman readable, printed referenceGPXMachine readable, handheld GPSFPLMachine readable, aircraft GPSFormat details:FormatNameVersionFile TypeEncodingOpen FormatRestricted AttributesExtensions AvailableExtensions UsedCSVComma Separated ValueN/ATextUTF-8 + BOMYesNoNoN/AGPXGPS Exchange Format1.1XMLUTF-8YesYesYesNoFPL(Garmin) Flight Plan1.0XMLUTF-8No (Vendor Specific)YesYesNoOutputs produced for each format:FormatEach WaypointEach RouteAll Waypoints (Only)All Routes (Only)Waypoints and Routes (Combined)CSVNoNoYesNo [1]NoGPXNoNoNo [1]No [1]YesFPLNoYesYesNoNoWhere 'All Waypoints (Only)' outputs are produced, waypoints will be sorted alphabetically.[1] These outputs can be produced but are intentionally excluded as they aren't used by the Air Unit. See thisGitLab issue 🛡for details.Output file namesOutput files use an internal naming convention for all formats:Export TypeFile Name (Pattern)File Name (Example)Each WaypointN/AN/AEach Route{route name}.ext01_ALPHA_TO_BRAVO.extAll Waypoints (Only)00_WAYPOINTS_{current date}.ext00_WAYPOINTS_2014_12_24.extAll Routes (Only)00_ROUTES_{current date}.ext00_ROUTES_2014_12_24.extWaypoints and Routes (Combined)00_NETWORK_{current date}.ext00_NETWORK_2014_12_24.extWhere.extis a relevant file extension for each format (i.e..csvfor CSV outputs).Output format - CSVNotes:for compatibility with Microsoft Excel, CSV outputs include the UTF-8 Byte Order Mark (BOM), which may cause issues
with other tools/applicationsCSV outputs use the first row as a column names headeroutputs produced for all routes use aroute_namecolumn to distinguish rows related to each routewaypoint.geometriescan optionally be included as separate latitude (Y) and longitude (X) columns in either:decimal degrees (latitude_dd,longitude_ddcolumns) - native formatdegrees, decimal minutes (latitude_ddm,longitude_ddmcolumns) - format used in aviationLimitations:all properties are encoded as strings, without type hints using extended CSV schemes etc.CSV outputs are not validatedOutput format - GPXNotes:GPX outputs are validated against the GPX XSD schema automaticallyLimitations:GPX metadata fields (author, last updated, etc.) are not currently populatedthe GPX comment field is set to thewaypoint.nameproperty only, as GPS devices used by the Air Unit only support
comments of upto 16 charactersOutput format - FPLNotes:FPL outputs are validated against a custom version of the GarminFPL XSD schemaautomaticallyroute names will use spaces instead of underscores in FPL files, as underscores aren't allowed in FPL route namesLimitations:thewaypoint.colocated_with,waypoint.last_accessed_at,waypoint.last_accessed_byandwaypoint.commentproperties are not included in FPL waypoint comments, as they are limited to 17 characters [1]underscores (_) characters are stripped from route nameswithinFPL files (rather than the namesofFPL
files), a local override is used to replace underscores with spaces () to work around this limitationFPL metadata fields (author, last updated, etc.) are not currently populated[1] This limit comes from the specific UI shown in the aircraft GPS used by the BAS Air Unit.FPL XML schemaA copy of the Garmin FPL XML/XSD schema,http://www8.garmin.com/xmlschemas/FlightPlanv1.xsd, is included in this
project to locally validate generated FPL outputs. This schema cannot be used for validation in its published form, as
it contains a number of invalid regular expressions. These regular expressions have been modified in the schema used
in this project, which hopefully match Garmin's intentions.In order to produce FPL files that match those produced by earlier processing scripts used by the BAS Air Unit, a
number of other changes have been made to the local version of the FPL schema. These include:removing the requirement for a<waypoints-table>element to be included in all FPL files (relevant to route FPLs)removing the requirement for all<waypoint>elements within<route>elements to be included in a<waypoint-table>element (as a consequence of the above)altering the regular expression used for the<country-name>element to allow the_charactersNote:It is hoped these local modifications will be removed in future through testing with the in-aircraft GPS.
See#12 🛡for more information.SetupRequirementsPython 3.9+libxml2 withxmlintbinary available on Pathread/write access to a suitable location for creating aWorkspace DirectoryNote:As of version 0.3.0, Windows is no longer a supported operating system for running this utility.Install Python packageIt is strongly recommended to install thePython Packagein a Python virtual environment:$ python -m venv /path/to/venv
$ source /path/to/venv/bin/activate
$ python -m pip install --upgrade pip
$ python -m pip install bas-air-unit-network-datasetDevelopmentLocal development environmentCheck out project:$ git clone https://gitlab.data.bas.ac.uk/MAGIC/air-unit-network-dataset.git
$ cd air-unit-network-datasetNote:If you do not have access to the BAS GitLab instance, clone from GitHub as a read-only copy instead.Poetryis used for managing the Python environment and dependencies.pyenvis strongly recommended to ensure the Python version is the same as the one
used in externally provisioned environments. This is currently3.9.18.$ pyenv install 3.9.18
$ pyenv local 3.9.18
$ poetry installEditorconfigFor consistency is strongly recommended to configure your IDE or other editor to use theEditorConfigsettings defined in.editorconfig.DependenciesDependency vulnerability checksTheSafetypackage is used to check dependencies against known vulnerabilities.WARNING!As with all security tools, Safety is an aid for spotting common mistakes, not a guarantee of secure code.
In particular this is using the free vulnerability database, which is updated less frequently than paid options.Checks are run automatically inContinuous Integration. To check locally:$ poetry run safety check --full-reportStatic security scanningRuff is configured to runBandit, a static analysis tool for Python.WARNING!As with all security tools, Bandit is an aid for spotting common mistakes, not a guarantee of secure code.
In particular this tool can't check for issues that are only be detectable when running code.lxmlpackage (bandit)Bandit identifies the use oflxmlclasses and methods as a security issue, specifically:Element to parse untrusted XML data is known to be vulnerable to XML attacksThe recommendation is to use asafeimplementation of an XML processor (defusedxml) that can avoid entity bombs and
other XML processing attacks. However,defusedxmldoes not offer all the methods we need and there does not appear
to be such another processor that does provide them.The main vulnerability this security issue relates to is processing user input that can't be trusted. This isn't really
applicable to this library directly, but rather to where it's used in implementing projects. I.e. if this library is
used in a service that accepts user input, an assessment must be made whether the input needs to be sanitised.Within this library itself, the only input that is processed is test records, all of which are assumed to be safe to
process.Code lintingRuffis used to lint and format Python files. Specific checks and config options are
set inpyproject.toml. Linting checks are run automatically inContinuous Integration.To check locally:$ poetry run ruff check src/ tests/
$ poetry run ruff format --check src/ tests/To format files:$ poetry run ruff format src/ tests/TestingBasic end-to-end tests are performed automatically inContinuous Integrationto check theTest Networkcan be processed via the Network Utility using thetests/create_outputs.py.$ poetry run python ./tests/create_outputs.py ./tests/resources/test-network/test-network.gpx ./tests/outTest outputs are compared against known good reference files intests/resources/test-network/reference-outputs/by
comparing checksums on file contents using thetests/compare_outputs.pyscript.$ poetry run python ./tests/compare_outputs.py ./tests/outContinuous IntegrationAll commits will trigger a Continuous Integration process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.DeploymentThe Air Unit Network utility is distributed as a Python package installed through Pip from thePyPiregistry.Source and binary packages are built and published automatically usingPoetryinContinuous Deployment.Note:Packages for non-tagged commits will use0.0.0as a version to indicate they're informal releases.To build the Python package manually:$ poetry buildTo publish the Python package to PyPi manually, you will need an API token for the BAS organisational PyPi account,
set as thePOETRY_PYPI_TOKEN_PYPIenvironment variable. Then run:$ poetry publish --username british-antarctic-surveyContinuous DeploymentAll commits will trigger a Continuous Deployment process using GitLab's CI/CD platform, configured in.gitlab-ci.yml.Releasesall releases 🛡latest release 🛡To create a release, create an issue using thereleaseissue template and follow the included checklist.FeedbackThis project is maintained by the BAS Mapping and Geographic Information Centre
(MAGIC), contactable at:magic@bas.ac.uk.LicenseCopyright (c) 2022 - 2023 UK Research and Innovation (UKRI), British Antarctic Survey.Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.