Repository: HypothesisWorks/hypothesis
Pull request: A new and terrifying age of continuous deployment is upon us
This pull request closes #408.

I am operating on the thesis that if something is worth doing then it's worth dialling up to 11, releasing the hounds, and then nuking it from orbit (after ensuring the hounds are safely out of the blast radius. Hypothesis does not condone cruelty to animals): All merges to master that are green on Travis (it does not check the other CI, but the other CI had to be green to merge in the first place so I think that's OK) and have any source changes from the last version will be immediately tagged for release and uploaded to pypi.

This PR contains two necessary knock on changes:

*  It adds some build steps that check that you have updated the version to something that is not tagged, and checks that both the the current date and the current version are present in the changelog (any checking beyond that is up to the reviewer).
* It updates the changelog and version to do a release of 3.7.1 today.

I think these changes do not make logical sense to separate (arguably the 3.7.1 change does, but I'd like to actually see this working in practice as soon as we merge it). I am happy to accept arguments to the contrary.

Why do this?

* Release automation is an intrinsic good - less scope for me to get things wrong.
* This will force us to be much better about changelog management than we have currently been.
* I have complete faith in our CI and review processes stopping anything that shouldn't be released making it into master. Therefore anything that is in master and is unreleased is basically awesome stuff that we are withholding from users. That seems like a thing we'd like to avoid.

On a person level, the other motivation for doing this is that I used to be *really* good about releasing early and releasing often, and due to a variety of reasons have stopped being so. This makes me sad, so I'd like to get back to it, but good intentions are rarely enough to do better: The way to do better is instead to bake it into the process.

A knock-on effect is that depending on the merge order, either #527 needs to be updated to talk about changelogs and versioning for all functionality change or this needs to be updated to include changes to the review list. My vote would be to merge this first and update #527.

Problem statement:
Better release automation
The current release process is a bit mickey mouse and involves me manually running pip commands. This is less than ideal.

It should be possible to set up Travis to build packages and upload them to pypi for us whenever there is a tag that matches the version number that passes its build.

If we get this working, the next step would be to look into also building wheels - we have CI builders on Windows, OSX and Linux, and the source release should be fine for anyone on something more obscure than one of those three. This doesn't really matter now while Hypothesis is pure Python, but I'd like it to stop being pure Python at some point - there are a number of core bits it would be nice to speed up with a C extension.

Hints from issue discussion:
I am generally 👍 for automation.

I am also in favour of cutting a 3.7.0 release soon, because we have a bunch of cool stuff on master – in particular, all your improvements to the shrinker – that nobody else is getting to use. Why should we keep all the good stuff to ourselves?

Assuming you agree, let’s use this stone to kill one bird and maim another. You cut the 3.7.0 release, write down what you need to do, and then I’ll look at writing some automation stuff for releases.

What do you think?

Commit message: Temporary revert so we can see the build jobs fail

Revert "Update version and changelog"

This reverts commit 98a3bae53d34030c89a9b131597ea9c27e80b5aa.

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 8303e9d54e..5706dc0208 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,6 +7,9 @@ env: PYTHONDONTWRITEBYTECODE=x
 os:
     - linux
 
+before_install:
+  - openssl aes-256-cbc -K $encrypted_39cb4cc39a80_key -iv $encrypted_39cb4cc39a80_iv -in deploy_key.enc -out deploy_key -d
+
 branches:
   only:
     - "master"
@@ -25,6 +28,8 @@ env:
 
     matrix:
         # Core tests that we want to run first.
+        - TASK=check-untagged
+        - TASK=check-changelog
         - TASK=documentation
         - TASK=lint
         - TASK=check-rst
@@ -63,3 +68,13 @@ notifications:
       - david@drmaciver.com
     on_success: never
     on_failure: change
+
+deploy:
+  provider: pypi
+  user: DRMacIver
+  password:
+    secure: T9tUyr9JG3KA8JprHUE8AAVaClUHHPHhiQRXC2/jHJv2EVEx2+EifkuR9sr5q1mCZ06g9v2+jhVNuaZyVuU9mmDKvJfADlPAagkCcgF/siuuJkzd3eIlELZHDtWPwKiZWCAD0Oj6yMdVB2w3/JTpJQhJx7HrL8EqbVdnN9YkMmI=
+  on:
+    branch: master
+    repo: HypothesisWorks/hypothesis-python
+    condition: make check-deploy-and-tag
diff --git a/Makefile b/Makefile
index 36cfeb5c84..d5a76b7864 100644
--- a/Makefile
+++ b/Makefile
@@ -84,6 +84,15 @@ format: $(PYFORMAT) $(ISORT)
 lint: $(FLAKE8)
 	$(FLAKE8) src tests --exclude=compat.py,test_reflection.py,test_imports.py,tests/py2,test_lambda_formatting.py --ignore=E731,E721
 
+check-untagged: $(BEST_PY3)
+	$(BEST_PY3) scripts/check-untagged.py
+
+check-changelog: $(BEST_PY3)
+	$(BEST_PY3) scripts/check-changelog.py
+
+check-deploy-and-tag: $(BEST_PY3)
+	$(BEST_PY3) scripts/should-deploy.py
+
 check-format: format
 	find src tests -name "*.py" | xargs $(TOOL_PYTHON) scripts/check_encoding_header.py
 	git diff --exit-code
diff --git a/deploy_key.enc b/deploy_key.enc
new file mode 100644
index 0000000000..e5b4c95091
Binary files /dev/null and b/deploy_key.enc differ
diff --git a/scripts/check-changelog.py b/scripts/check-changelog.py
new file mode 100644
index 0000000000..477dc0217c
--- /dev/null
+++ b/scripts/check-changelog.py
@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+sys.path.append(os.path.dirname(__file__)) # noqa
+
+import hypothesistooling as tools
+from datetime import datetime
+
+
+if __name__ == '__main__':
+    changelog = tools.changelog()
+
+    if "\n%s - " % (tools.__version__,) not in changelog:
+        print(
+            "The current version (%s) isn't mentioned in the changelog" % (
+                tools.__version__,))
+        sys.exit(1)
+
+    when = datetime.utcnow().strftime("%Y-%m-%d")
+
+    if when not in changelog:
+        print((
+            "The current date (%s) isn't mentioned in the changelog. "
+            "Remember this will be released as soon as you merge to master!"
+        ) % (when,))
diff --git a/scripts/check-untagged.py b/scripts/check-untagged.py
new file mode 100755
index 0000000000..7bfe53b11c
--- /dev/null
+++ b/scripts/check-untagged.py
@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+sys.path.append(os.path.dirname(__file__)) # noqa
+
+import hypothesistooling as tools
+
+
+if __name__ == '__main__':
+    if tools.__version__ in tools.tags():
+        if tools.has_source_changes(tools.__version__):
+            print("Has code changes from existing released version %s" % (
+                tools.__version__,
+            ))
+            print(
+                "This means you should update src/hypothesis/version.py "
+                "to a new version before merging to master. Don't forget "
+                "to update the changelog too!"
+            )
+            sys.exit(1)
diff --git a/scripts/hypothesistooling.py b/scripts/hypothesistooling.py
new file mode 100644
index 0000000000..d45b95eb82
--- /dev/null
+++ b/scripts/hypothesistooling.py
@@ -0,0 +1,79 @@
+import subprocess
+import os
+
+
+def current_branch():
+    return subprocess.check_output([
+        "git", "rev-parse", "--abbrev-ref", "HEAD"
+    ]).decode('ascii').strip()
+
+
+def tags():
+    result = [t.decode('ascii') for t in subprocess.check_output([
+        "git", "tag"
+    ]).split(b"\n")]
+    assert len(set(result)) == len(result)
+    return set(result)
+
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+SRC = os.path.join(ROOT, "src")
+
+
+__version__ = None
+
+
+with open(os.path.join(ROOT, "src/hypothesis/version.py")) as o:
+    exec(o.read())
+
+assert __version__ is not None
+
+
+def latest_version():
+    versions = []
+
+    for t in tags():
+        assert t == t.strip()
+        parts = t.split(".")
+        if len(parts) != 3:
+            continue
+        try:
+            v = tuple(map(int, parts))
+        except ValueError:
+            continue
+        try:
+            versions.append((v, t))
+        except ValueError:
+            pass
+
+    _, latest = max(versions)
+
+    assert latest in tags()
+    return latest
+
+
+def changelog():
+    with open(os.path.join(ROOT, "docs", "changes.rst")) as i:
+        return i.read()
+
+
+DEVNULL = open(os.devnull)
+
+
+def has_source_changes(version):
+    return subprocess.call([
+        "git", "diff", "--exit-code", version, SRC,
+    ], stdout=DEVNULL, stderr=DEVNULL) != 0
+
+
+def git(*args):
+    subprocess.check_call(("git",) + args)
+
+
+def create_tag():
+    assert __version__ not in tags()
+    git("config", "user.name", "Travis CI on behalf of David R. MacIver")
+    git("config", "user.email", "david@drmaciver.com")
+    git("config", "core.sshCommand", "ssh -i deploy_key")
+    git("tag", __version__)
+    git("push", "--tags")
diff --git a/scripts/should-deploy.py b/scripts/should-deploy.py
new file mode 100644
index 0000000000..f38fe73e53
--- /dev/null
+++ b/scripts/should-deploy.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+sys.path.append(os.path.dirname(__file__)) # noqa
+
+import hypothesistooling as tools
+
+
+if __name__ == '__main__':
+    branch = tools.current_branch()
+
+    if branch != "master":
+        print("Not deploying due to non-master branch %s" % (branch,))
+        sys.exit(1)
+
+    last_release = tools.latest_version()
+
+    if not tools.has_source_changes(last_release):
+        print("Not deploying due to no source changes")
+        sys.exit(1)
+
+    print("Current version: %s. Latest released version: %s" % (
+        tools.__version__, last_release
+    ))
+
+    print("Looks good to release! Pushing the tag now.")
+    tools.create_tag()
+    sys.exit(0)


Code review comments:
diff_hunk: @@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+sys.path.append(os.path.dirname(__file__)) # noqa
+
+import hypothesistooling as tools
+from datetime import datetime
+
+
+if __name__ == '__main__':
+    changelog = tools.changelog()
+
+    if "\n%s - " % (tools.__version__,) not in changelog:
+        print(
+            "The current version (%s) isn't mentioned in the changelog" % (
+                tools.__version__,))
+        sys.exit(1)
+
+    when = datetime.utcnow().strftime("%Y-%m-%d")
+
+    if when not in changelog:
path: scripts/check-changelog.py
text: @user1:
potential failure condition: what do we want to happen if the CI run happens "just" after crossing a day boundary?

@author:
I've added some wiggle room so it allows any date within +/-1 hour of now.

diff_hunk: @@ -0,0 +1,79 @@
+import subprocess
+import os
+
+
+def current_branch():
+    return subprocess.check_output([
+        "git", "rev-parse", "--abbrev-ref", "HEAD"
+    ]).decode('ascii').strip()
+
+
+def tags():
+    result = [t.decode('ascii') for t in subprocess.check_output([
+        "git", "tag"
+    ]).split(b"\n")]
+    assert len(set(result)) == len(result)
+    return set(result)
+
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
path: scripts/hypothesistooling.py
text: @user1:
this is a nasty failure if we ever move the scripts directory

`git rev-parse --show-toplevel` may be a better fit, and the following should be enough to get it

```
ROOT = subprocess.check_output(["git", "rev-parse",
    "--show-toplevel"]).decode('ascii').strip()
```

@author:
I think it's OK, because if we move scripts we'll fail at reading version.py later, but I'm happy to change it.

diff_hunk: @@ -0,0 +1,79 @@
+import subprocess
+import os
+
+
+def current_branch():
+    return subprocess.check_output([
+        "git", "rev-parse", "--abbrev-ref", "HEAD"
+    ]).decode('ascii').strip()
+
+
+def tags():
+    result = [t.decode('ascii') for t in subprocess.check_output([
+        "git", "tag"
+    ]).split(b"\n")]
+    assert len(set(result)) == len(result)
+    return set(result)
+
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+SRC = os.path.join(ROOT, "src")
+
+
+__version__ = None
+
+
+with open(os.path.join(ROOT, "src/hypothesis/version.py")) as o:
+    exec(o.read())
+
+assert __version__ is not None
+
+
+def latest_version():
+    versions = []
+
+    for t in tags():
+        assert t == t.strip()
+        parts = t.split(".")
+        if len(parts) != 3:
+            continue
+        try:
+            v = tuple(map(int, parts))
+        except ValueError:
+            continue
+        try:
+            versions.append((v, t))
+        except ValueError:
+            pass
+
+    _, latest = max(versions)
+
+    assert latest in tags()
+    return latest
+
+
+def changelog():
+    with open(os.path.join(ROOT, "docs", "changes.rst")) as i:
+        return i.read()
+
+
+DEVNULL = open(os.devnull)
path: scripts/hypothesistooling.py
text: @user1:
Alternatively, `subprocess.DEVNULL`?

@author:
Huh. I failed to find that.