Package: ds-best-practice-example
ds-best-practice-exampleShowcase best practices for structuring a simple DS-related Python package/application.If you notice any best practice missing/incorrect, please do let me know and I will update them.This example package will fit a linear regression on your input data. It comes with thediabetesdata as a toy data.Note that this example comes with many tools/components to help your development, but do enable/disable them based on your needs.The recommend minimum tools I usually go for in most of my projects arePoetry- for Python package managementPre-commit- for formating and lintinghydra- for package configurationslogging- for logging runtime informationPytest- for unit testingThe additional tools areMkdocs- for documentation generationMkdocstrings- for automatic API generationTox- for testing under multiple environmentsCodecov- for code coverage reportGitHub Actions- for CI/CDGitHub Pages- for documentation hostingRelated linksDocumentation:https://cheeyeelim.github.io/ds-best-practice-exampleGitHub:https://github.com/cheeyeelim/ds-best-practice-examplePyPI:https://pypi.org/project/ds-best-practice-example/Free software: BSD-3-ClauseHow to install this package?The easiest way is to install it with pip.pipinstallds-best-practice-exampleHow to run this package?You run this package via both command line and Python.Note that all configurations are specified viahydrain thisconfig yaml.Please refer to theexample notebookfor more details.(1) Run via command linepoetryrunezml(2) Run via Pythonimporthydrafromezml.dataimportprepare_datafromezml.modelimporttrain_modelfromezml.diagnosticimportdiagnose_model# Use Hydra config with Compose APIhydra.initialize(version_base=None,config_path="../input/conf")cfg=hydra.compose(config_name="config",overrides=["data.file_path=../input/data/diabetes.csv","model.file_path=../output/model/linreg_diabetes.joblib"])# Train linear regression with default dataX_train,X_test,y_train,y_test=prepare_data(cfg)model=train_model(cfg,X_train,y_train)diagnose_model(X_test,y_test,model)How to develop this package further?Update codes as needed.Usually I create and test codes in Jupyter notebook (undernotebookfolder) before manually adapting over to standard Python scripts.(Test locally) Test that codes are working as intended.Test locally (all in one go)poetry run toxInternallytoxwill run unit testing, document generation and build tests.Test locally (one by one)poetry run pytestfor unit testingpoetry run mike deploy vtest -m "test doc build" --ignorepoetry run mike delete vtest -m "remove doc build" --ignorepoetry run mkdocs serveto see docs locallypoetry buildpoetry run twine check dist/*to test buildsTest on cloudNo need to do anythingFollow later steps to push the codes to GitHub to trigger tests, as this repo has GitHub Workflows defined (in.github/workflows)Runpre-commitby committing codes.git add .git commit -m "a message"Resolve any errors frompre-commitmanually.Bump the version number upE.g. for a patch,poetry run bump2version patchRerun git add and commit to commit codes.Once happy with everything,git pushthe codes to cloud repo.GitHub Actions will be automatically triggered for testing and staging.Wait for GitHub Actions to complete, then check for a published package athttps://test.pypi.org/project/ds-best-practice-example/Once all are done, triggerreleasebuild by tagging a commit withv*version number.A documentation will be automatically generated athttps://cheeyeelim.github.io/ds-best-practice-exampleThe package will be built and published toPyPI.Done!How to create this project from scratch?You can create it from scratch by following the steps below.The steps assumedcookiecutter,pre-commitandpoetryhas been installed. Otherwise follow their respective installation instructions to install them.# Setup project folder interactivelycookiecutterhttps://github.com/cheeyeelim/cookiecutter-pypackage.git# Setup pre-commitpre-commitinstall# Update pyproject.toml with required packages# Install Python packagespoetryinstall-Etest-Edev-EdocCreditsThis package was created withCookiecutterand thecheeyeelim/cookiecutter-pypackageproject template.