Package: pds.registry-sweepers
Registry SweepersThis package provides supplementary metadata generation for registry documents, which is required for registry-api to function correctly, and for common user queries. Execution is idempotent and should be scheduled on a recurring basis.ComponentsRepairKitThe repairkit sweeper applies idempotent transformations to targeted subsets of properties, for example ensuring that all properties expected to have array-like values are in fact arrays (as opposed to single-element arrays being flattened to strings during harvest).  Documents are processed based on whether theirops:Provenance/ops:registry_sweepers_repairkit_versionis up-to-date relative to the sweeper codebase.ProvenanceThe provenance sweeper generates metadata for linking each version-superseded product with the versioned product which supersedes it.  The value of the successor is stored in theops:Provenance/ops:superseded_byproperty.  This property will not be set for the latest version of any product.AncestryThe ancestry sweeper generates membership metadata for each product, i.e. which bundle lidvids and which collection lidvids reference a given product. These values will be stored in propertiesops:Provenance/ops:parent_bundle_identifierandops:Provenance/ops:parent_collection_identifier, respectively.Accepts environment variables to tune performance, primarily trading increased runtime duration for reduced peak memory usage.Developer QuickstartPrerequisitesDependenciesPython >=3.9Environment VariablesPROV_CREDENTIALS={"admin": "admin"}  // OpenSearch username/password
PROV_ENDPOINT=https://localhost:9200  // OpenSearch host url and port
LOGLEVEL - an integer log level or anycase string matching a python log level like `INFO` (optional - defaults to `INFO`))
DEV_MODE=1  // disables host verificationWith--legacy-syncoption, you also need the list of the cross-cluster-search node configured to access all the node's OpensSearch domains:CCS_CONN=naif-prod-ccs,rms-prod,sbnumd-prod-ccs,geo-prod-ccs,atm-prod-ccs,sbnpsi-prod-ccs,ppi-prod-ccs,img-prod-ccsUse the connection aliases found in the 'Connections' tab of the Engineering Node OpenSearch Domain on AWS.https://us-west-2.console.aws.amazon.com/aos/home?region=us-west-2#opensearch/domains/en-prod?tabId=ccsAfter cloning the repository, and setting the repository root as the current working directory install the package withpip install -e .The wrapper script for the suite of components may be run withpython ./docker/sweepers_driver.pyAlternatively, registry-sweepers may be built from itsDockerfilewithdocker image build --file ./docker/Dockerfile .and run as a container, providing those same environment variables when running the container.PerformanceRough BenchmarksWhen run against the production OpenSearch instance with ~1.1M products, no cross-cluster remotes, and (only) ~1k multi-version products, from a local development machine, the runtime is ~20min on first run and ~12min subsequently.  It appears that OpenSearch optimizes away no-op update calls, resulting in significant speedup despite the fact that registry-sweepers reprocesses metadata from scratch, every run.The overwhelming bottleneck ops are the O(docs_count) db writes in ancestry.Code of ConductAll users and developers of the NASA-PDS software are expected to abide by ourCode of Conduct. Please read this to ensure you understand the expectations of our community.DevelopmentTo develop this project, use your favorite text editor, or an integrated development environment with Python support, such asPyCharm.ContributingFor information on how to contribute to NASA-PDS codebases please take a look at ourContributing guidelines.InstallationInstall in editable mode and with extra developer dependencies into your virtual environment of choice:pip install --editable '.[dev]'Configure thepre-commithooks:pre-commit install
pre-commit install -t pre-push
pre-commit install -t prepare-commit-msg
pre-commit install -t commit-msgThese hooks check code formatting and also aborts commits that contain secrets such as passwords or API keys. However, a one time setup is required in your global Git configuration. Seethe wiki entry on Git Secretsto learn how.PackagingTo isolate and be able to re-produce the environment for this package, you should use aPython Virtual Environment. To do so, run:python -m venv venvThen exclusively usevenv/bin/python,venv/bin/pip, etc.If you havetoxinstalled and would like it to create your environment and install dependencies for you run:tox --devenv <name you'd like for env> -e devDependencies for development are specified as thedevextras_requireinsetup.cfg; they are installed into the virtual environment as follows:pip install --editable '.[dev]'All the source code is in a sub-directory undersrc.TestsThis section describes testing for your package.A complete "build" including test execution, linting (mypy,black,flake8, etc.), and documentation build is executed via:toxUnit testsYour project should have built-in unit tests, functional, validation, acceptance, etc., tests.For unit testing, check out theunittestmodule, built into Python 3.Tests objects should be in packagestestmodules or preferably in project 'tests' directory which mirrors the project package structure.Our unit tests are launched with command:pytestIf you want your tests to run automatically as you make changes start uppytestin watch mode with:ptwBuildpip install wheel
python setup.py sdist bdist_wheelPublicationNASA PDS packages can publish automatically using theRoundup Action, which leverages GitHub Actions to perform automated continuous integration and continuous delivery. A default workflow that includes the Roundup is provided in the.github/workflows/unstable-cicd.yamlfile. (Unstable here means an interim release.)Manual PublicationCreate the package:python setup.py bdist_wheelPublish it as a Github release.Publish on PyPI (you need a PyPI account and configure$HOME/.pypirc):pip install twine
twine upload dist/*Or publish on the Test PyPI (you need a Test PyPI account and configure$HOME/.pypirc):pip install twine
twine upload --repository testpypi dist/*CI/CDThe template repository comes with our two "standard" CI/CD workflows,stable-cicdandunstable-cicd. The unstable build runs on any push tomain(± ignoring changes to specific files) and the stable build runs on push of a release branch of the formrelease/<release version>. Both of these make use of our GitHub actions build step,Roundup. Theunstable-cicdwill generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with av0.0.0-SNAPSHOTrelease (see NASA-PDS/roundup-action#56 for specifics).