Package: registry-sweepers
PDS Template Repository for PythonThis is the template repository for PDS's Python projects.This repository aims at being a base for new python repositories used in PDS. It guides developers to ease the initialization of a project and recommends preferred options to standardize developments and ease maintenance. Simply click theUse this templatebutton ↑ (or usethis hyperlink).🏃 Getting Started With This TemplateSee our wiki page for more info on setting up your new repo. You can remove this section once you have completed the necessary start-up steps.https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#creating-a-new-repo👉 Important!You must assign the teams as mentioned on the wiki page above! At a minimum, these are:TeamPermission@NASA-PDS/pds-software-committerswrite@NASA-PDS/pds-software-pmcadmin@NASA-PDS/pds-operationsadminMy ProjectThis is the XYZ that does this, that, and the other thing for the Planetary Data System.Please visit our website at:https://nasa-pds.github.io/pds-my-projectIt has useful information for developers and end-users.PrerequisitesInclude any system-wide requirements (brew install,apt-get install,yum install, …)Python 3should be used regardless asPython 2 reached end-of-life on January 1st, 2020.User QuickstartInstall with:pip install my_pds_moduleIf possible, make it so that your program works out of the box without any additional configuration—but see theConfigurationsection for details.To execute, run:(put your run commands here)Code of ConductAll users and developers of the NASA-PDS software are expected to abide by ourCode of Conduct. Please read this to ensure you understand the expectations of our community.DevelopmentTo develop this project, use your favorite text editor, or an integrated development environment with Python support, such asPyCharm.ContributingFor information on how to contribute to NASA-PDS codebases please take a look at ourContributing guidelines.InstallationInstall in editable mode and with extra developer dependencies into your virtual environment of choice:pip install --editable '.[dev]'Configure thepre-commithooks:pre-commit install
pre-commit install -t pre-push
pre-commit install -t prepare-commit-msg
pre-commit install -t commit-msgThese hooks check code formatting and also aborts commits that contain secrets such as passwords or API keys. However, a one time setup is required in your global Git configuration. Seethe wiki entry on Git Secretsto learn how.PackagingTo isolate and be able to re-produce the environment for this package, you should use aPython Virtual Environment. To do so, run:python -m venv venvThen exclusively usevenv/bin/python,venv/bin/pip, etc.If you havetoxinstalled and would like it to create your environment and install dependencies for you run:tox --devenv <name you'd like for env> -e devDependencies for development are specified as thedevextras_requireinsetup.cfg; they are installed into the virtual environment as follows:pip install --editable '.[dev]'All the source code is in a sub-directory undersrc.You should update thesetup.cfgfile with:name of your modulelicense, default apache, update if neededdescriptiondownload url, when you release your package on github add the url herekeywordsclassifiersinstall_requires, add the dependencies of you packageextras_require, add the development Dependencies of your packageentry_points, when your package can be called in command line, this helps to deploy command lines entry points pointing to scripts in your packageFor the packaging details, seehttps://packaging.python.org/tutorials/packaging-projects/as a reference.ConfigurationIt is convenient to use ConfigParser package to manage configuration. It allows a default configuration which can be overwritten by the user in a specific file in their environment. Seehttps://pymotw.com/2/ConfigParser/For example:candidates = ['my_pds_module.ini', 'my_pds_module.ini.default']
found = parser.read(candidates)LogsYou should not useprint()vin the purpose of logging information on the execution of your code. Depending on where the code runs these information could be redirected to specific log files.To make that work, start each Python file with:importlogginglogger=logging.getLogger(__name__)To log a message:logger.info("my message")In yourmainroutine, include:logging.basicConfig(level=logging.INFO)to get a basic logging system configured.ToolingThedevextras_requireincluded in the template repo installsblack,flake8(plus some plugins), andmypyalong with default configuration for all of them. You can run all of these (and more!) with:tox -e lintCode StyleSo that your code is readable, you should comply with thePEP8 style guide. Our code style is automatically enforced in viablackandflake8. See theTooling sectionfor information on invoking the linting pipeline.❗Important note for template users❗
The includedpre-commit configuration fileexecutesflake8(along withmypy) across the entiresrcfolder and not only on changed files. If you're converting a pre-existing code base over to this template that may result in a lot of errors that you aren't ready to deal with.You can instead executeflake8only over a diff of the current changes being made by modifying thepre-commitentryline:entry: git diff -u | flake8 --diffOr you can change thepre-commitconfig soflake8is only called on changed files which match a certain filtering criteria:-   repo: local
    hooks:
    -   id: flake8
        name: flake8
        entry: flake8
        files: ^src/|tests/
        language: systemRecommended LibrariesPython offers a large variety of libraries. In PDS scope, for the most current usage we should use:LibraryUsageconfigparsermanage and parse configuration filesargparsecommand line argument documentation and parsingrequestsinteract with web APIslxmlread/write XML filesjsonread/write JSON filespyyamlread/write YAML filespystachegenerate files from templatesSome of these are built into Python 3; others are open source add-ons you can include in yourrequirements.txt.TestsThis section describes testing for your package.A complete "build" including test execution, linting (mypy,black,flake8, etc.), and documentation build is executed via:toxUnit testsYour project should have built-in unit tests, functional, validation, acceptance, etc., tests.For unit testing, check out theunittestmodule, built into Python 3.Tests objects should be in packagestestmodules or preferably in project 'tests' directory which mirrors the project package structure.Our unit tests are launched with command:pytestIf you want your tests to run automatically as you make changes start uppytestin watch mode with:ptwIntegration/Behavioral TestsOne should use thebehave packageand push the test results to "testrail".See an example inhttps://github.com/NASA-PDS/pds-doi-service#behavioral-testing-for-integration--testingDocumentationYour project should useSphinxto build its documentation. PDS' documentation template is already configured as part of the default build. You can build your projects docs with:python setup.py build_sphinxYou can access the build files in the following directory relative to the project root:build/sphinx/html/Buildpip install wheel
python setup.py sdist bdist_wheelPublicationNASA PDS packages can publish automatically using theRoundup Action, which leverages GitHub Actions to perform automated continuous integration and continuous delivery. A default workflow that includes the Roundup is provided in the.github/workflows/unstable-cicd.yamlfile. (Unstable here means an interim release.)Manual PublicationCreate the package:python setup.py bdist_wheelPublish it as a Github release.Publish on PyPI (you need a PyPI account and configure$HOME/.pypirc):pip install twine
twine upload dist/*Or publish on the Test PyPI (you need a Test PyPI account and configure$HOME/.pypirc):pip install twine
twine upload --repository testpypi dist/*CI/CDThe template repository comes with our two "standard" CI/CD workflows,stable-cicdandunstable-cicd. The unstable build runs on any push tomain(± ignoring changes to specific files) and the stable build runs on push of a release branch of the formrelease/<release version>. Both of these make use of our GitHub actions build step,Roundup. Theunstable-cicdwill generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with av0.0.0-SNAPSHOTrelease (see NASA-PDS/roundup-action#56 for specifics).