Package: geostore
GeostoreLINZ central storage, management and access solution for important geospatial datasets. Developed byLand Information New Zealand.PrerequisitesGeostore VPCA Geostore VPC must exist in your AWS account before deploying this application. AT LINZ, VPCs are
managed internally by the IT team. If you are deploying this application outside LINZ, you will need
to create a VPC with the following tags:"ApplicationName": "geostore""ApplicationLayer": "networking"You can achieve this by adding thenetworking_stack(infrastructure/networking_stack.py)intoapp.pybefore deployment as a dependency ofapplication_stack(infrastructure/application_stack.py).Verify infrastructure settingsThis infrastructure by default includes some Toitū Te Whenua-/LINZ-specific parts, controlled by
settings in cdk.json. To disable these, simply remove the context entries or set them tofalse.
The settings are:enableLDSAccess: if true, gives LINZ Data Service/Koordinates read access to the storage bucket.enableOpenTopographyAccess: if true, gives OpenTopography read access to the storage bucket.Development setupOne-time setup which generally assumes that you're in the project directory.CommonInstall DockerConfigure Docker:Add yourself to the "docker" group:sudo usermod --append --groups=docker "$USER"Log out and back in to enable the new groupUbuntuInstallnvm:cd"$(mktemp--directory)"wgethttps://raw.githubusercontent.com/nvm-sh/nvm/master/install.shecho'b674516f001d331c517be63c1baeaf71de6cbb6d68a44112bf2cff39a6bc246a install.sh'|sha256sum--check&&bashinstall.shInstallPoetry:cd"$(mktemp--directory)"wgethttps://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.pyecho'b35d059be6f343ac1f05ae56e8eaaaebb34da8c92424ee00133821d7f11e3a9c install-poetry.py'|sha256sum--check&&python3install-poetry.pyInstallPyenv:sudoapt-getupdate
sudoapt-getinstall--no-install-recommendsbuild-essentialcurllibbz2-devlibffi-devliblzma-devlibncurses5-devlibreadline-devlibsqlite3-devlibssl-devlibxml2-devlibxmlsec1-devllvmmaketk-devwgetxz-utilszlib1g-devcd"$(mktemp--directory)"wgethttps://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installerecho'3aa49f2b3b77556272a80a01fe44d46733f4862dbbbc956002dc944c428bebd8 pyenv-installer'|sha256sum--check&&bashpyenv-installerEnable the above by adding the following to your~/.bashrc:if[[-e"${HOME}/.local/bin"]]thenPATH="${HOME}/.local/bin:${PATH}"fi# nvm <https://github.com/nvm-sh/nvm>if[[-d"${HOME}/.nvm"]]thenexportNVM_DIR="${HOME}/.nvm"# shellcheck source=/dev/null[[-s"${NVM_DIR}/nvm.sh"]]&&."${NVM_DIR}/nvm.sh"# shellcheck source=/dev/null[[-s"${NVM_DIR}/bash_completion"]]&&."${NVM_DIR}/bash_completion"fi# Pyenv <https://github.com/pyenv/pyenv>if[[-e"${HOME}/.pyenv"]]thenPATH="${HOME}/.pyenv/bin:${PATH}"eval"$(pyenvinit--path)"eval"$(pyenvinit-)"eval"$(pyenvvirtualenv-init-)"fiConfigure Docker:Add yourself to the "docker" group:sudo usermod --append --groups=docker "$USER"Log out and back in to enable the new groupInstall project Node.js:nvm installRun./reset-dev-env.bash --allto install packages.Enable the dev environment:. activate-dev-env.bash.Optional: EnableDependabot alerts by email. (This
is optional since it currently can't be set per repository or organisation, so it affects any
repos where you have access to Dependabot alerts.)Installaws-azure-login.Re-run./reset-dev-env.bashwhen packages change. One easy way to use it pretty much seamlessly is
to run it before every workday, with a crontab entry like this template:HOME='/home/USERNAME'
0 2 * * 1-5 export PATH="${HOME}/.pyenv/shims:${HOME}/.pyenv/bin:${HOME}/.poetry/bin:/root/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/run/current-system/sw/bin" && cd "PATH_TO_GEOSTORE" && ./reset-dev-env.bash --allReplaceUSERNAMEandPATH_TO_GEOSTOREwith your values, resulting in something like this:HOME='/home/jdoe'
0 2 * * 1-5 export PATH="${HOME}/.pyenv/shims:${HOME}/.pyenv/bin:${HOME}/.poetry/bin:/root/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/run/current-system/sw/bin" && cd "${HOME}/dev/geostore" && ./reset-dev-env.bash --allRe-run. activate-dev-env.bashin each shell.NixRunnix-shell.Optional: Install andconfiguredirenvanddirenv allow .to load the Nix shell whenever youcdinto the project.Restart yournix-shellwhen packages change.When setting up the project SDK point it to.run/python, which is a symlink to the latest Nix
shell Python executable.OptionalEnableDependabot alerts by email. (This is optional
since it currently can't be set per repository or organisation, so it affects any repos where you
have access to Dependabot alerts.)AWS Infrastructure deploymentConfigure AWSGet AWS credentials (see:https://www.npmjs.com/package/aws-azure-login) for 12 hours:aws-azure-login--no-prompt--profile=<AWS-PROFILE-NAME>Environment variablesGEOSTORE_ENV_NAME:set deployment environment. For your personal development stack: set
GEOSTORE_ENV_NAME to your username.exportGEOSTORE_ENV_NAME="$USER"Other values used by CI pipelines include: prod, nonprod, ci, dev or any string without spaces.
Default: test.RESOURCE_REMOVAL_POLICY:determines if resources containing user content like Geostore
Storage S3 bucket or application database tables will be preserved even if they are removed
from stack or stack is deleted. Supported values:DESTROY: destroy resource when removed from stack or stack is deleted (default)RETAIN: retain orphaned resource when removed from stack or stack is deletedGEOSTORE_SAML_IDENTITY_PROVIDER_ARN:SAML identity provider AWS ARN.Bootstrap CDK (only once per profile)cdk--profile=<AWS-PROFILE-NAME>bootstrapaws://unknown-account/ap-southeast-2Deploy CDK stackcdk--profile=<AWS-PROFILE-NAME>deploy--allOnce comfortable with CDK you can add--require-approval=neverabove to deploy
non-interactively.If youexport AWS_PROFILE=<AWS-PROFILE-NAME>you won't need the--profile=<AWS-PROFILE-NAME>arguments above.DevelopmentAdding or updating Python dependenciesTo add a development-only package:poetry add --dev PACKAGE='*'To add a production package:Install the package usingpoetry add --optional PACKAGE='*'.Put the package in alphabetical order within the list.Mention the package in the relevant lists in[tool.poetry.extras].Make sure to update packages separately from adding packages. Basically, follow this process
before runningpoetry add, and do the equivalent when updating Node.js packages or changing
Docker base images:Check out a new branch on top of origin/master:git checkout -b update-python-packages origin/master.Update the Python packages:poetry update. The rest of the steps are only necessary if this
step changes poetry.lock. Otherwise you can just change back to the original branch and delete
"update-python-packages".Commit, push and create pull request.Check out the branch where you originally wanted to runpoetry add.Rebase the branch onto the package update branch:git rebase update-python-packages.At this point anypoetry addcommands should not result in any package updates other than those
necessary to fulfil the new packages' dependencies.Rationale: Keeping upgrades and other packages changes apart is useful when reading/bisecting
history. It also makes code review easier.When there's a merge conflict in poetry.lock, first check whether either or both commits contain a
package upgrade:If neither of them do, simplygit checkout --ours -- poetry.lock && poetry lock --no-update.If one of them does, check out that file (git checkout --ours -- poetry.lockorgit checkout --theirs -- poetry.lock) and runpoetry lock --no-updateto regeneratepoetry.lockwith the current package versions.If both of them do, manually mergepoetry.lockand runpoetry lock --no-update.Rationale: This should avoid accidentally down- or upgrading when resolving a merge conflict.Update the code coverage minimum in pyproject.toml and the badge above on branches which increase
it.Rationale: By updating this continuously we avoid missing test regressions in new branches.Upgrading Python versionTo minimise the chance of discrepancies between environments it is important to run the same (or as
close as possible) version of Python in the development environment, in the pipeline, and in
deployed instances. At the moment the available versions are constrained by the following:TheUbuntu packagesused in theDockerfileTheAWS base imagesused asLambda runtimesThepyenv versionsused forlocal developmentThesupported Poetry versionsused for alldependenciesWhen updating Python versions you have to check that all of the above can be kept at the same minor
version, and ideally at the same patch level.Running testsPrerequisites:Authenticated to a profile which has access to a deployed Geostore.To launch full test suite, runpytest.DebuggingTo start debugging at a specific line, insertimport ipdb; ipdb.set_trace().To debug a test run, add--capture=noto thepytestarguments. You can also automatically start
debugging at a test failure point with--pdb --pdbcls=IPython.terminal.debugger:Pdb.Upgrading CI runnerjobs.<job_id>.runs-onin .github sets the runner type per job. We should make sure all of these use the latest specific
("ubuntu-YY.MM" as opposed to "ubuntu-latest") Ubuntu LTS version, to make sure the version changes
only when we're ready for it.GitHub Actions cache clearingTo throw away the current cache (for example in case of a cache corruption), simply change theCACHE_SEEDrepository "secret",
for example to the current timestamp (date +%s). Subsequent jobs will then ignore the existing
cache.