Repository: Galileo-Galilei/kedro-mlflow (language: python)

Issue:
Improve development and release workflow
The development workflow can be updated to adress some current issues: 

- update contributing process (PR to develop):
    - [x] _add a contribution section to mention contributors: @kaemo @akruszewski I feel your name should be mentioned as your contribution grow more and more -> _I have added a "hall of fame" badges with the number of commit of the most active people of the repo. Is it sataisfying? What other solutions would you prefer?_
    - [X] ~force the CI to fail under 100% test coverage: several recent PR have broken coverage (mainly #9 but I had spotted breaking the 100% coverage before)~ _(EDIT 2020-09-26)For now, I don't add the ` --cov-fail-under=100` flag to the "test" step in the build workflow because it will malke all CI fail until we add all the missing tests. I mark it as resolved and add it after the next release. I think that the codecov bot gives far enough information to take decision to merge A PR with the implications in mind._
    - [x] update the coverage to a dedicated third party (codecov? I don't have any preference here, an idea?) to identify more easily the coverage breaking PR.
 
- update release process both in the documentation and in the CI (PR to master from develop):
     - [x] use an action to create a release in the github repo (if possible taking beeing fed automatically by the changelog, maybe [create-release](https://github.com/actions/create-release)
    - [x] fix the deployment to pypi and move to a public action (maybe [gh-action-pypi-publish
](https://github.com/pypa/gh-action-pypi-publish))
   - [x] change the trigger of the worflow (closing a PR even if dismissed currently triggers the pipeline: it should only be triggered when a merge on master is completed).

Do you have any other suggestion @kaemo and @akruszewski ?

Improve development and release workflow
The development workflow can be updated to adress some current issues: 

- update contributing process (PR to develop):
    - [x] _add a contribution section to mention contributors: @kaemo @akruszewski I feel your name should be mentioned as your contribution grow more and more -> _I have added a "hall of fame" badges with the number of commit of the most active people of the repo. Is it sataisfying? What other solutions would you prefer?_
    - [X] ~force the CI to fail under 100% test coverage: several recent PR have broken coverage (mainly #9 but I had spotted breaking the 100% coverage before)~ _(EDIT 2020-09-26)For now, I don't add the ` --cov-fail-under=100` flag to the "test" step in the build workflow because it will malke all CI fail until we add all the missing tests. I mark it as resolved and add it after the next release. I think that the codecov bot gives far enough information to take decision to merge A PR with the implications in mind._
    - [x] update the coverage to a dedicated third party (codecov? I don't have any preference here, an idea?) to identify more easily the coverage breaking PR.
 
- update release process both in the documentation and in the CI (PR to master from develop):
     - [x] use an action to create a release in the github repo (if possible taking beeing fed automatically by the changelog, maybe [create-release](https://github.com/actions/create-release)
    - [x] fix the deployment to pypi and move to a public action (maybe [gh-action-pypi-publish
](https://github.com/pypa/gh-action-pypi-publish))
   - [x] change the trigger of the worflow (closing a PR even if dismissed currently triggers the pipeline: it should only be triggered when a merge on master is completed).

Do you have any other suggestion @kaemo and @akruszewski ?



Discussion / hints:
I absolutely agree with all the points. 

I'm not sure what coverage tool would be best to use. I know *CircleCI* is used often for Python open-source projects for many CI/CD tasks. It allows running tests on different platforms, creating coverage reports, building packages and deploying them so it might be a good shout. It could also cover releasing on PyPI. That would, of course, require rewriting our GitHub actions completely and moving them to CircleCI.
Let's add this to the issues to adress when we're having a call. 

For the record, I think we should be pragmatic here and keep using github actions on the short term (I guess it will be easier/faster to add the actions I suggested to the existing github workflows rather to rewrite the CI even if it is not difficult, but please go ahead if you really want it :wink: ). CircleCI is definitely one of the most used, let's keep it on the list of potential tools.
I added the following items to the to do list:
- [X] add templates (inspired from kedro's) for the issues / PR
- [X] some README improvements and badges for clarity
- [X] change the matrix in CI to test also on windows and macos
The create-release candidate pipeline is failing because of a github deprecation for security issues: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/. Updating the vsoch/pull-request-action to 1.0.12 should be sufficient to fix it.
The publish pipeline is failing too for the same reason. 
- Read the docs build is failing, likely due to the pandas 1.2.0rc version. I pin pandas an numpy in docs requirements to ensure it works (it also breaks all rebuild of former version!)
- codecov upload fails inconsistently. I add a token hoping it will make upload more stable.
I absolutely agree with all the points. 

I'm not sure what coverage tool would be best to use. I know *CircleCI* is used often for Python open-source projects for many CI/CD tasks. It allows running tests on different platforms, creating coverage reports, building packages and deploying them so it might be a good shout. It could also cover releasing on PyPI. That would, of course, require rewriting our GitHub actions completely and moving them to CircleCI.
Let's add this to the issues to adress when we're having a call. 

For the record, I think we should be pragmatic here and keep using github actions on the short term (I guess it will be easier/faster to add the actions I suggested to the existing github workflows rather to rewrite the CI even if it is not difficult, but please go ahead if you really want it :wink: ). CircleCI is definitely one of the most used, let's keep it on the list of potential tools.
I added the following items to the to do list:
- [X] add templates (inspired from kedro's) for the issues / PR
- [X] some README improvements and badges for clarity
- [X] change the matrix in CI to test also on windows and macos
The create-release candidate pipeline is failing because of a github deprecation for security issues: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/. Updating the vsoch/pull-request-action to 1.0.12 should be sufficient to fix it.
The publish pipeline is failing too for the same reason. 
- Read the docs build is failing, likely due to the pandas 1.2.0rc version. I pin pandas an numpy in docs requirements to ensure it works (it also breaks all rebuild of former version!)
- codecov upload fails inconsistently. I add a token hoping it will make upload more stable.

Fix patch:
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 4948a041..e9b38d26 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -18,6 +18,7 @@ formats:
 python:
   version: 3.6
   install:
-    - requirements: requirements/docs_requirements.txt
-    - method: setuptools
+    - method: pip
       path: .
+      extra_requirements:
+       - docs
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 4caea64a..e81814ae 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -2,11 +2,14 @@
 
 The current workflow is the following:
 
-1. Open an issue to describe your feature request or your bug fix with detailed explanation of what you want to achieve.
+1. Open an issue to describe your feature request or your bug fix with a detailed explanation of what you want to achieve.
 2. Fork the repo
 3. Develop locally:
     - Install the precommit file (`pip install pre-commit`, then `pre-commit install`)
     - Create a branch based on the develop branch (``git checkout -b <prefix-branchname> develop``)
+    - Create a conda environment (conda create -n <your-env-name> python==3.7)
+    - Activate this environment (`conda activate <your-env-name>`)
+    - Install the extra dependencies for tests (`pip install kedro-mlflow[tests]`)
     - Apply your changes
     - Run pre-commit (black linting, flake8 errors, isort with ``pre-commit run``)
 4. Submit your changes:
@@ -42,4 +45,4 @@ The current workflow is the following:
 4. Checkout the [publish workflow](https://github.com/Galileo-Galilei/kedro-mlflow/actions?query=workflow%3Apublish) to see if:
     - The package has been uploaded on PyPI sucessfully
     - The changes have been merged back to develop
-5. If the pipeline has failed, please raise an issue to correct the CI, and ensure merge on develop "by hand"".
+5. If the pipeline has failed, please raise an issue to correct the CI, and ensure merge on develop manually.
diff --git a/kedro_mlflow/framework/cli/cli.py b/kedro_mlflow/framework/cli/cli.py
index 4c4b5418..c4b41666 100644
--- a/kedro_mlflow/framework/cli/cli.py
+++ b/kedro_mlflow/framework/cli/cli.py
@@ -56,7 +56,7 @@ def commands():
 @commands.command(name="mlflow", cls=KedroClickGroup)
 def mlflow_commands():
     """Use mlflow-specific commands inside kedro project."""
-    pass
+    pass  # pragma: no cover
 
 
 @mlflow_commands.command()
diff --git a/requirements/requirements.txt b/requirements.txt
similarity index 100%
rename from requirements/requirements.txt
rename to requirements.txt
diff --git a/requirements/docs_requirements.txt b/requirements/docs_requirements.txt
deleted file mode 100644
index d8040e24..00000000
--- a/requirements/docs_requirements.txt
+++ /dev/null
@@ -1,6 +0,0 @@
-sphinx==3.4.2
-recommonmark==0.7.1
-sphinx_rtd_theme==0.5.1
-sphinx-markdown-tables==0.0.15
-pandas>=1.0.0, <1.3.0 # avoid to make readthedocs load rc version
-numpy>=1.0.0, <1.19.0 # bug on windows
diff --git a/setup.py b/setup.py
index 234b630d..821e312e 100644
--- a/setup.py
+++ b/setup.py
@@ -15,8 +15,7 @@ def _parse_requirements(path, encoding="utf-8"):
 
 
 # get the dependencies and installs
-base_requirements = _parse_requirements("requirements/requirements.txt")
-test_requirements = _parse_requirements("requirements/test_requirements.txt")
+base_requirements = _parse_requirements("requirements.txt")
 
 
 # Get the long description from the README file
@@ -36,8 +35,27 @@ def _parse_requirements(path, encoding="utf-8"):
     packages=find_packages(exclude=["docs*", "tests*"]),
     setup_requires=["setuptools_scm"],
     include_package_data=True,
-    tests_require=test_requirements,
     install_requires=base_requirements,
+    extras_require={
+        "docs": [
+            "sphinx==3.4.2",
+            "recommonmark==0.7.1",
+            "sphinx_rtd_theme==0.5.1",
+            "sphinx-markdown-tables==0.0.15",
+            "pandas>=1.0.0, <1.3.0",  # avoid to make readthedocs load rc version
+            "numpy>=1.0.0, <1.19.0",  # bug on windows for numpy 1.19.0->1.19.4
+        ],
+        "tests": [
+            "pytest>=5.4.0, <7.0.0",
+            "pytest-cov>=2.8.0, <3.0.0",
+            "pytest-lazy-fixture>=0.6.0, <1.0.0",
+            "pytest-mock>=3.1.0, <4.0.0",
+            "scikit-learn>=0.23.0, <0.25.0",
+            "flake8==3.8.4",  # ensure consistency with pre-commit
+            "black==20.8b1",  # pin black version because it is not compatible with a pip range (because of non semver version number)
+            "isort==5.7.0",  # ensure consistency with pre-commit
+        ],
+    },
     author="Galileo-Galilei",
     entry_points={
         "kedro.project_commands": [


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 52e1b5be..08688ac2 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -28,17 +28,15 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -r requirements/requirements.txt
-        pip install -r requirements/test_requirements.txt
+        pip install -e .[tests]
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
         flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics --exclude kedro_mlflow/template/project/run.py
         # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics --exclude kedro_mlflow/template/project/run.py
+        flake8 . --count --max-complexity=10 --max-line-length=127 --statistics --exclude kedro_mlflow/template/project/run.py
     - name: Test with pytest and generate coverage report
       run: |
-        pip install -e .  # required for some tests which launches commands via subprocess
         pytest --cov=./ --cov-report=xml
     - name: Upload coverage report to Codecov
       uses: codecov/codecov-action@v1
diff --git a/requirements/test_requirements.txt b/requirements/test_requirements.txt
deleted file mode 100644
index f5b9fe0b..00000000
--- a/requirements/test_requirements.txt
+++ /dev/null
@@ -1,8 +0,0 @@
-pytest>=5.4.0, <7.0.0
-pytest-cov>=2.8.0, <3.0.0
-pytest-lazy-fixture>=0.6.0, <1.0.0
-pytest-mock>=3.1.0, <4.0.0
-scikit-learn>=0.23.0, <0.25.0
-flake8==3.8.4
-black==20.8b1  # pin black version because it is not compatible with a pip range (because of non semver version number)
-isort==5.7.0
diff --git a/tests/test_utils.py b/tests/test_utils.py
index abbbb351..74dfb7cb 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -5,7 +5,6 @@
 
 from kedro_mlflow.utils import (
     KEDRO_VERSION,
-    KedroContextError,
     _already_updated,
     _get_project_globals,
     _is_kedro_project,