Package: np-validator
np-validatornp-validatoris just a simple Python cli/package that validates data sources for the neuropixel pipline using templated workflows.Quick overviewAPIAuto-generated stepsBy defaultrun_validationcan be run on a list of
filepaths. A list ofValidationStepwill be auto-generated if a valid project name is supplied.If no project name is supplied project name will be "default".fromnp_validatorimportrun_validationfilepaths=["some/sort/of/path/prefix/uuid-maybe.mapping.pkl","some/sort/of/path/prefix/uuid-maybe.behavior.pkl","this/will/be/ignored/uuid-maybe.replay.pkl",]results=run_validation(filepaths,project="default")Manually generating validation stepsfromnp_validatorimportProcessor,Validator,ValidationStep,run_validation# make a basic filesize validatorfs_validator=Validator(name="meets_filesize_threshold",args={"threshold":10,},)# add validator to a validation stepvalidation_step_0=ValidationStep(path_suffix=".mapping.pkl",validators=[fs_validator,],)# make a validation step with a processor# processors convert the data source from it's basic state, a filepath, to an easier to use object# unpickle, unpickles an arbitrary filepath into a python objectunpickler=Processor(name="unpickle",)# has_dict_key, checks if a dict-like interface has a key at pathsession_uuid_validator=Validator(name="has_dict_key",args={"path":["session_uuid",],})# assembling it all together as a validation stepvalidation_step_1=ValidationStep(path_suffix=".behavior.pkl",processor=unpickler,validators=[session_uuid_validator,],)# running a validationfilepaths=["some/sort/of/path/prefix/uuid-maybe.mapping.pkl","some/sort/of/path/prefix/uuid-maybe.behavior.pkl","this/will/be/ignored/uuid-maybe.replay.pkl",]results=run_validation(filepaths,[validation_step_0,validation_step_1,],)Updating autogenerated stepsfromnp_validatorimportProcessor,Validator,ValidationStep,autogenerate_validation_steps,update_project_validation_steps# autogenerated steps are organized by projectsvalidation_step_0=ValidationStep(path_suffix=".mapping.pkl",validators=[Validator(name="meets_filesize_threshold",args={"threshold":10,},),],)validation_step_1=ValidationStep(path_suffix=".behavior.pkl",processor=Processor(name="unpickle",),validators=[Validator(name="has_dict_key",args={"path":["session_uuid",],}),],)# updating project validation steps overwrites the previous stepsupdate_project_validation_steps("pretest",# project name[validation_step_0,validation_step_1],)# to append, get the current steps with autogenerate_validation_steps and append to itcurrent=autogenerate_validation_steps("default")updated=current+[validation_step_0,validation_step_1]update_project_validation_steps("default",# project nameupdated,# append new steps)DocumentationFor more detailed documentation on using this package please refer to thedocs.ContributingProcessorTo add a new processor, add the function tonp_validator/processors.py. Ideally create a test for it intests. Run thetests* to help ensure that no regressions have been introduced. Each processor is expected to have one required argument which is expected to be a string filepath.ValidatorTo add a new validator, add the function tonp_validator/validators.py. Ideally create a test for it intests. Run thetests* to help ensure that no regressions have been introduced.*Tests automatically run on pull_request.Installationpipinstall-Unp-validatoror install withPoetrypoetryaddnp-validatorMakefile usageMakefilecontains a lot of functions for faster development.1. Download and remove PoetryTo download and install Poetry run:makepoetry-downloadTo uninstallmakepoetry-remove2. Install all dependencies and pre-commit hooksInstall requirements:makeinstallPre-commit hooks coulb be installed aftergit initviamakepre-commit-install3. CodestyleAutomatic formatting usespyupgrade,isortandblack.makecodestyle# or use synonymmakeformattingCodestyle checks only, without rewriting files:makecheck-codestyleNote:check-codestyleusesisort,blackanddarglintlibraryUpdate all dev libraries to the latest version using one comandmakeupdate-dev-deps4. Code securitymakecheck-safetyThis command launchesPoetryintegrity checks as well as identifies security issues withSafetyandBandit.makecheck-safety5. Type checksRunmypystatic type checkermakemypy6. Tests with coverage badgesRunpytestmaketest7. All lintersOf course there is a command torulerun all linters in one:makelintthe same as:maketest&&makecheck-codestyle&&makemypy&&makecheck-safety8. Dockermakedocker-buildwhich is equivalent to:makedocker-buildVERSION=latestRemove docker image withmakedocker-removeMore informationabout docker.9. CleanupDelete pycache filesmakepycache-removeRemove package buildmakebuild-removeDelete .DS_STORE filesmakedsstore-removeRemove .mypycachemakemypycache-removeOr to remove all above run:makecleanup📈 ReleasesYou can see the list of available releases on theGitHub Releasespage.We followSemantic Versionsspecification.We useRelease Drafter. As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.List of labels and corresponding titlesLabelTitle in Releasesenhancement,feature🚀 Featuresbug,refactoring,bugfix,fix🔧 Fixes & Refactoringbuild,ci,testing📦 Build System & CI/CDbreaking💥 Breaking Changesdocumentation📝 Documentationdependencies⬆️ Dependencies updatesYou can update it inrelease-drafter.yml.GitHub creates thebug,enhancement, anddocumentationlabels for you. Dependabot creates thedependencieslabel. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.Additonal informationInitializing a new repoInitialize your codeInitializegitinside your repo:cdnp-validator&&gitinitIf you don't havePoetryinstalled run:makepoetry-downloadInitialize poetry and installpre-commithooks:makeinstall
makepre-commit-installRun the codestyle:makecodestyleUpload initial code to GitHub:gitadd.
gitcommit-m":tada: Initial commit"gitbranch-Mmain
gitremoteaddoriginhttps://github.com/np_validator/np-validator.git
gitpush-uoriginmainSet up botsSet upDependabotto ensure you have the latest dependencies.Set upStale botfor automatic issue closing.PoetryWant to know more about Poetry? Checkits documentation.Details about PoetryPoetry'scommandsare very intuitive and easy to learn, like:poetry add numpy@latestpoetry run pytestpoetry publish --buildetcBuilding and releasing your packageBuilding a new version of the application contains steps:Bump the version of your packagepoetry version <version>. You can pass the new version explicitly, or a rule such asmajor,minor, orpatch. For more details, refer to theSemantic Versionsstandard.Make a commit toGitHub.Create aGitHub release.And... publish 🙂poetry publish --build🎯 What's nextWell, that's up to you 💪🏻. I can only recommend the packages and articles that helped me.Typeris great for creating CLI applications.Richmakes it easy to add beautiful formatting in the terminal.Pydantic– data validation and settings management using Python type hinting.Logurumakes logging (stupidly) simple.tqdm– fast, extensible progress bar for Python and CLI.IceCreamis a little library for sweet and creamy debugging.orjson– ultra fast JSON parsing library.Returnsmakes you function's output meaningful, typed, and safe!Hydrais a framework for elegantly configuring complex applications.FastAPIis a type-driven asynchronous web framework.Articles:Open Source Guides.A handy guide to financial support for open sourceGitHub Actions Documentation.Maybe you would like to addgitmojito commit names. This is really funny. 😄🚀 FeaturesDevelopment featuresSupports forPython 3.9and higher.Poetryas the dependencies manager. See configuration inpyproject.tomlandsetup.cfg.Automatic codestyle withblack,isortandpyupgrade.Ready-to-usepre-commithooks with code-formatting.Type checks withmypy; docstring checks withdarglint; security checks withsafetyandbanditTesting withpytest.Ready-to-use.editorconfig,.dockerignore, and.gitignore. You don't have to worry about those things.Deployment featuresGitHubintegration: issue and pr templates.Github Actionswith predefinedbuild workflowas the default CI/CD.Everything is already set up for security checks, codestyle checks, code formatting, testing, linting, docker builds, etc withMakefile. More details inmakefile-usage.Dockerfilefor your package.Always up-to-date dependencies with@dependabot. You will onlyenable it.Automatic drafts of new releases withRelease Drafter. You may see the list of labels inrelease-drafter.yml. Works perfectly withSemantic Versionsspecification.Open source community featuresReady-to-usePull Requests templatesand severalIssue templates.Files such as:LICENSE,CONTRIBUTING.md,CODE_OF_CONDUCT.md, andSECURITY.mdare generated automatically.Stale botthat closes abandoned issues after a period of inactivity. (You will onlyneed to setup free plan). Configuration ishere.Semantic Versionsspecification withRelease Drafter.