Repository: theislab/cellrank (language: python)

Issue:
Outline branching structure in `CONTRIBUTING.rst`
## Description

Using the `master`-`dev` approach, we need to ensure using a consistent branching structure.

### In a nutshell

* Prior to a release, a release branch `release/X.X.X` should be branched which then merged into `cellrank@master`, `cellrank@dev`.
* For hotfixes, a branch `hotfix/X.X.X` is created based off of `cellrank/master` and merged into both `cellrank@master`, `cellrank@dev`.
* On the release branches we can add release note etc. prior to the actual release. This allows us to continue working on `cellrank@dev`.
* Release merges should be either merge commits, *not* squash merge. This is both helpful for `git blame` and prevents `cellrank@master` and  `cellrank@dev` from diverging.


Discussion / hints:
Agreed, let's stick to this. 
Will give you my thoughts on this during next week, but overall looks good to me.

Just for reference, here's the link: https://nvie.com/posts/a-successful-git-branching-model/
I think that the hotfix stuff might be a bit too complicated for our repo, but am willing to give it a try.

Yes, exactly what I had envisioned!

Fix patch:
diff --git a/.github/workflows/deployment.yml b/.github/workflows/deployment.yml
new file mode 100644
index 000000000..2a8adae58
--- /dev/null
+++ b/.github/workflows/deployment.yml
@@ -0,0 +1,71 @@
+name: Deployment
+
+on:
+    push:
+        branches: [master]
+        tags: [v*]
+    workflow_dispatch:
+        inputs:
+            reason:
+                description: Reason for the workflow dispatch.
+                required: true
+                default: release
+
+jobs:
+    deploy:
+        if: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.reason == 'release') || (github.event_name == 'push' && startsWith(github.ref, 'refs/tags')) }}
+        runs-on: ubuntu-latest
+        steps:
+        -   uses: actions/checkout@v2
+        -   name: Set up Python 3.8
+            uses: actions/setup-python@v2
+            with:
+                python-version: 3.8
+
+        -   name: Install pip dependencies
+            run: |
+                python -m pip install --upgrade pip
+                pip install build
+
+        # this will fail if the last commit is not tagged
+        -   name: Build a binary wheel and a source tarball
+            run: |
+                python -m build --sdist --wheel --outdir dist/
+
+        -   name: Publish CellRank on PyPI
+            uses: pypa/gh-action-pypi-publish@master
+            with:
+                user: __token__
+                password: ${{ secrets.PYPI_TOKEN }}
+                skip_existing: true
+                verbose: true
+
+    sync-branches:
+        if: github.event_name == 'workflow_dispatch'
+        needs: deploy
+        runs-on: ubuntu-latest
+        steps:
+        -   uses: actions/checkout@v2
+            with:
+                token: ${{ secrets.TOWNCRIER_TOKEN }}
+
+        -   name: Extract the branch name
+            id: vars
+            run: |
+                echo ::set-output name=branch::${GITHUB_REF#refs/*/}
+
+        -   name: Merge release into dev
+            uses: stefanzweifel/git-auto-commit-action@v4
+            with:
+                branch: dev
+                commit_user_name: CI
+                commit_message: ${{ format('[auto][ci skip] Merge branch '{0}' into dev', steps.vars.output.branch) }}
+                commit_options: --no-ff
+
+        -   name: Merge release into master
+            uses: stefanzweifel/git-auto-commit-action@v4
+            with:
+                branch: master
+                commit_user_name: CI
+                commit_message: ${{ format('[auto][ci skip] Merge branch '{0}' into master', steps.vars.output.branch) }}
+                commit_options: --no-ff
diff --git a/.github/workflows/news.yml b/.github/workflows/news.yml
index fc1743e7a..d98fd0d31 100644
--- a/.github/workflows/news.yml
+++ b/.github/workflows/news.yml
@@ -2,7 +2,7 @@ name: News
 
 on:
     pull_request_target:
-        branches: [master, dev]
+        branches: [dev]
         types: [closed]
 
 jobs:
@@ -14,6 +14,7 @@ jobs:
         -   uses: actions/checkout@v2
             with:
                 token: ${{ secrets.TOWNCRIER_TOKEN }}
+
         -   name: Set up Python 3.8
             uses: actions/setup-python@v2
             with:
@@ -24,6 +25,7 @@ jobs:
                 python -m pip install --upgrade pip
                 pip install towncrier rstcheck requests
                 pip install -e'.'
+
         -   name: Generate news fragment
             env:
                 PR_NUMBER: ${{ github.event.number }}
@@ -35,14 +37,14 @@ jobs:
             id: auto-commit-action
             uses: stefanzweifel/git-auto-commit-action@v4
             with:
-                commit_message: ${{ format('[auto][ci skip] Generate news fragment (#{0})', github.event.number) }}
                 branch: ${{ github.base_ref }}
                 file_pattern: docs/source/release/changelog/*.rst
                 commit_user_name: CI
+                commit_message: ${{ format('[auto][ci skip] Generate news fragment (#{0})', github.event.number) }}
                 skip_dirty_check: false
 
         -   name: Update dev release notes
-            if: ${{ steps.auto-commit-action.outputs.changes_detected == 'true' && contains(github.base_ref, 'dev') }}
+            if: steps.auto-commit-action.outputs.changes_detected == 'true'
             run: |
                 rm -f docs/source/release/notes-dev.rst
                 towncrier build --yes --version dev
@@ -54,11 +56,11 @@ jobs:
                 git checkout -- docs/source/release/changelog/
 
         -   name: Commit dev release notes
-            if: ${{ steps.auto-commit-action.outputs.changes_detected == 'true' && contains(github.base_ref, 'dev') }}
+            if: steps.auto-commit-action.outputs.changes_detected == 'true'
             uses: stefanzweifel/git-auto-commit-action@v4
             with:
-                commit_message: ${{ format('[auto][ci skip] Update dev release notes (#{0})', github.event.number) }}
                 branch: ${{ github.base_ref }}
                 file_pattern: docs/source/release/notes-dev.rst
+                commit_message: ${{ format('[auto][ci skip] Update dev release notes (#{0})', github.event.number) }}
                 commit_user_name: CI
                 skip_dirty_check: false
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
new file mode 100644
index 000000000..8dd136a20
--- /dev/null
+++ b/.github/workflows/release.yml
@@ -0,0 +1,72 @@
+name: Release
+
+on: create
+
+jobs:
+    release:
+        if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
+        runs-on: ubuntu-latest
+        steps:
+        -   uses: actions/checkout@v2
+            with:
+                token: ${{ secrets.TOWNCRIER_TOKEN }}
+
+        -   name: Set up Python 3.8
+            uses: actions/setup-python@v2
+            with:
+                python-version: 3.8
+
+        -   name: Extract the tag
+            id: vars
+            run: |
+                echo ::set-output name=tag::${GITHUB_REF##*/}
+
+        -   name: Check tag validity
+            env:
+                VERSION: ${{ steps.vars.output.tag }}
+            run: |
+                grep "^v[[:digit:]]\.[[:digit:]]\.[[:digit:]]$" <<< "$VERSION" || (echo "Invalid version: '$VERSION'" && exit 42)
+
+        -   name: Install pip dependencies
+            run: |
+                python -m pip install --upgrade pip
+                pip install towncrier rstcheck bump2version
+                pip install -e'.'
+
+        -   name: Generate release notes
+            run: |
+                towncrier build --yes
+                rm -f docs/source/release/notes-dev.rst
+                towncrier build --yes --version dev
+                rstcheck -r docs/source/release
+
+        -   name: Commit release notes
+            uses: stefanzweifel/git-auto-commit-action@v4
+            with:
+                file_pattern: docs/source/release/**
+                commit_user_name: CI
+                commit_message: ${{ format('[auto][ci skip] Generate {0} release notes', steps.vars.output.tag) }}
+                skip_dirty_check: false
+
+        -   name: Bump the version
+            # the part (patch) doesn't matter when supplying --new-version
+            env:
+                VERSION: ${{ steps.vars.output.tag }}
+            run: |
+                bump2version patch --no-commit --no-tag --verbose --new-version "${VERSION/v/}"
+
+        -   name: Commit version bump
+            uses: stefanzweifel/git-auto-commit-action@v4
+            with:
+                file_pattern: setup.py cellrank/__init__.py .bumpversion.cfg
+                commit_user_name: CI
+                commit_message: ${{ format('[auto][ci skip] Release {0}', steps.vars.output.tag) }}
+                tagging_message: ${{ steps.vars.output.tag }}
+                skip_dirty_check: false
+
+        -   name: Invoke test workflow
+            uses: benc-uk/workflow-dispatch@v1
+            with:
+                workflow: Test
+                token: ${{ secrets.RELEASE_DISPATCH_TOKEN }}
+                inputs: '{ "reason": "release" }'
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 13506a1a8..9b4177af9 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -7,6 +7,7 @@ Table of Contents
 
   - `Setup`_
   - `Codebase structure`_
+  - `Branching structure`_
 
 - `Adding new features`_
 
@@ -71,6 +72,17 @@ The CellRank is structured as follows:
   `documentation <https://cellrank.readthedocs.io/en/latest/auto_examples/index.html>`__, see `Tutorials/examples`_
   for more information.
 
+Branching structure
+~~~~~~~~~~~~~~~~~~~
+We use sightly modified branching structured described
+`here <https://nvie.com/posts/a-successful-git-branching-model/>`_, in short:
+
+- ``feature/X``: for features, enhancements, etc.; it is based off the ``dev`` branch and squash merged into it again.
+- ``fix/X``: for bug fixes; is based off the ``dev`` branch and squash merged into it again.
+- ``release/vX.X.X``: prior to making a release; it is based off ``dev`` and then merged (using merge commit) into
+  ``master`` and ``dev``, after bumping the version and making release notes.
+  Both are usually done through CI, see `Making a new release`_.
+
 Adding new features
 ~~~~~~~~~~~~~~~~~~~
 We welcome every contribution to CellRank, whether it's a new kernel/estimator/model,
@@ -199,20 +211,30 @@ a new release has been made and an automatic PR will be made to
 Extra care has to be taken when updating runtime dependencies - this is not automatically picked up by Bioconda
 and a separate PR with the updated ``recipe.yaml`` will have to be made.
 
-To make creating new release as easy as possible, we use ``bump2version``, which can be installed as::
+Easiest way to create a new release it to create a branch named ``release/vX.X.X`` and push it onto GitHub. The CI
+will take care of the following:
+
+- create the new release notes (and empty the ``dev`` release notes)
+- bump the version and create a new tag
+- run tests on the ``release/vX.X.X`` branch
+- publish on PyPI after the tests have passed
+- merge ``release/vX.X.X`` into ``master`` and ``dev``
+
+Alternatively, it's possible to create a new release using ``bump2version``, which can be installed as::
 
     pip install bump2version
 
-Depending on what part of the version you want to update, you can run::
+Depending on what part of the version you want to update, you can run on ``master``::
 
     bump2version {major,minor,patch}
 
-By default, this will create a new tag, automatically update the ``__version__`` wherever necessary and commit the
-changes. Afterwards, you can just push the changes to upstream by running::
+By default, this will create a new tagged commit, automatically update the ``__version__`` wherever necessary.
+Afterwards, you can just push the changes to upstream by running::
 
     git push --atomic <branch> <tag>
 
-or set ``push.followtags=true`` in your git config and do a regular ``git push``.
+or set ``push.followtags=true`` in your git config and do a regular ``git push``. In this case, CI will not run
+create any release notes, run tests or do any merges.
 
 Tutorials/examples
 ~~~~~~~~~~~~~~~~~~
diff --git a/README.rst b/README.rst
index e8c5eb2d2..3bf7ff983 100644
--- a/README.rst
+++ b/README.rst
@@ -89,7 +89,7 @@ CellRank was developed in collaboration between the `Theislab`_ and the `Peerlab
     :target: https://bioconda.github.io/recipes/cellrank/README.html
     :alt: Bioconda
 
-.. |CI| image:: https://img.shields.io/github/workflow/status/theislab/cellrank/CI/master
+.. |CI| image:: https://img.shields.io/github/workflow/status/theislab/cellrank/Test/dev
     :target: https://github.com/theislab/cellrank/actions
     :alt: CI
 
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 1a8be147b..c3aceaf9a 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -107,7 +107,7 @@ CellRank was developed in collaboration between the `Theislab`_ and the `Peerlab
     :target: https://bioconda.github.io/recipes/cellrank/README.html
     :alt: Bioconda
 
-.. |CI| image:: https://img.shields.io/github/workflow/status/theislab/cellrank/CI/master
+.. |CI| image:: https://img.shields.io/github/workflow/status/theislab/cellrank/Test/dev
     :target: https://github.com/theislab/cellrank/actions
     :alt: CI
 


Test patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/test.yml
similarity index 63%
rename from .github/workflows/ci.yml
rename to .github/workflows/test.yml
index 64687e940..9a0070ba4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/test.yml
@@ -1,21 +1,24 @@
-name: CI
+name: Test
 
 on:
     schedule:
     -   cron: 00 00 * * 1  # every Monday at 00:00
     push:
         branches: [master, dev]
-        tags: [v*]
     pull_request:
         branches: [master, dev]
+    workflow_dispatch:
+        inputs:
+            reason:
+                description: Reason for the workflow dispatch.
+                required: true
+                default: release
 
 jobs:
     lint:
         runs-on: ubuntu-latest
         steps:
         -   uses: actions/checkout@v2
-            with:
-                fetch-depth: 0
         -   name: Set up Python 3.8
             uses: actions/setup-python@v2
             with:
@@ -53,8 +56,6 @@ jobs:
 
         steps:
         -   uses: actions/checkout@v2
-            with:
-                fetch-depth: 0
         -   name: Set up Python ${{ matrix.python }}
             uses: actions/setup-python@v2
             with:
@@ -70,6 +71,7 @@ jobs:
             id: pip-cache
             run: |
                 echo "::set-output name=dir::$(pip cache dir)"
+
         -   name: Restore pip cache
             uses: actions/cache@v2
             with:
@@ -99,15 +101,15 @@ jobs:
             env:
                 CODECOV_NAME: ${{ matrix.os }}-${{ matrix.python }}-{{ matrix.use_slepc }}
             run: |
-                codecov --no-color --required --flags unittests
+                codecov --no-color --required --flags unittest
 
-    regenerate:
-        # TODO(michalk8): modify so that notebooks regenerate with the appropriate ref
-        if: github.event_name == 'push' && startsWith(github.ref, 'refs/head')
+    regenerate-notebooks:
+        # TODO(michalk8): version also cellrank_notebooks?
+        if: (github.event_name == 'push' && github.ref == 'refs/heads/dev') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
         needs: [lint, test]
         runs-on: ubuntu-latest
         steps:
-        -   name: Rebuild tutorials
+        -   name: Request notebooks regeneration
             uses: peter-evans/repository-dispatch@v1
             with:
                 token: ${{ secrets.ACCESS_TOKEN }}
@@ -115,48 +117,14 @@ jobs:
                 event-type: rebuild
                 client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'
 
-    deploy:
-        if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
+    release:
+        if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.reason == 'release' }}
         needs: [lint, test]
         runs-on: ubuntu-latest
         steps:
-        -   uses: actions/checkout@v2
-            with:
-                token: ${{ secrets.TOWNCRIER_TOKEN }}
-                fetch-depth: 0
-        -   name: Set up Python 3.8
-            uses: actions/setup-python@v2
-            with:
-                python-version: 3.8
-
-        -   name: Install dependencies
-            run: |
-                python -m pip install --upgrade pip
-                pip install build towncrier rstcheck
-                pip install -e'.'
-
-        -   name: Generate release notes
-            run: |
-                towncrier build --yes
-                rm -f docs/source/release/notes-dev.rst
-                towncrier build --yes --version dev
-                rstcheck -r docs/source/release
-        -   name: Commit release notes
-            uses: stefanzweifel/git-auto-commit-action@v4
-            with:
-                commit_message: '[auto][ci skip] Generate release notes'
-                file_pattern: docs/source/release/**
-                commit_user_name: CI
-                skip_dirty_check: false
-
-        -   name: Build a binary wheel and a source tarball
-            run: |
-                python -m build --sdist --wheel --outdir dist/
-
-        -   name: Publish package on PyPI
-            uses: pypa/gh-action-pypi-publish@master
+        -   name: Invoke deployment workflow
+            uses: benc-uk/workflow-dispatch@v1
             with:
-                user: __token__
-                password: ${{ secrets.PYPI_TOKEN }}
-                skip_existing: true
-                verbose: true
+                workflow: Deployment
+                token: ${{ secrets.RELEASE_DISPATCH_TOKEN }}
+                inputs: '{ "reason": "release" }'