Repository: certbot/certbot

Issue:
Improve the process of deploying the Windows installer
Currently it requires manually copying it from Azure and creating a GitHub release which is cumbersome and error prone. We've already made a mistake doing this once during the release process.

To fix this, I think we have a few options:

1. Do a bit of threat modeling and convince ourselves we're fine giving Azure the permissions it needs to create GitHub releases.
2. [If Azure makes the change in a week and a half to allow Azure to create GitHub releases with its existing permissions](https://github.com/microsoft/azure-pipelines-tasks/issues/9754#issuecomment-626352963), do that. If it needs new permissions, see 1.
3. Write a script we run locally that pulls the files from Azure and publishes them to GitHub.

My personal preference between these options from most to least preferred are: 2, 1, 3

EDIT: Thinking slightly further into the future, I think we need to do some form of option 3. See https://github.com/certbot/certbot/issues/8049#issuecomment-663168809.


Discussion:
Thinking ahead a bit, in the medium term Azure Pipelines won't be able to create the release without manual interaction because the Windows installer will need to be signed and I personally think storing the private key for our code signing certificate in CI is a bad idea.

In this case, I think we either need to:

1. Write a script we run locally that pulls the files from Azure, signs them, and publishes them to GitHub.
2. Have Azure Pipelines pause while building the release, we download the installer, sign it, upload it to Azure, and it publishes the release to GitHub.

@adferrand, do you have an opinion between these two approaches or the problem in general?
In general to define the manual steps involved in finalizing the publications for a release, I will focus on a solution that is suitable for all assets that are going to be published. I think we should avoid dedicated and separated procedures for each asset.

If Azure allows a pause and a conditional continue of a pipeline, and that this approach can be used for Windows installer, snaps and dockers, I am in favor of 2, becauses it gives most of the hard work to an automated pipeline. If it is not possible, 1. will be sufficient as it fulfills the primary purpose of the CD: having a set of deployable assets that are fully tested and safe to be published.
That's a very good point about trying to design a solution that allows us to consolidate everything into one procedure. I have some ideas and am going to label this issue as blocked on me writing those ideas up somewhere since I think it'll influence the approach we want to take here.
I started to write a more general release automation issue, but found myself really wanting to talk about this specific issue. That's not to say I don't think we shouldn't work towards the goal of having a single, well automated release process and I think that's something we should keep in mind when working on this stuff and it's something I have ideas for, but I think the way we realistically build such a process with the resources we have is slowly over time.

To say a little about what I'm thinking more generally to help satisfy concerns there, I think we should automate everything in CI we reasonably can as I think it will provide the most consistent and reliable automation. There are some things I don't think we should do in CI though. For instance, I think storing the private key for our code signing certificate for the Windows installer in CI is a bad idea. More generally, I think we will have actions that will need to be run on a developer's machine to access credentials stored in a more secure manner (on things like smart cards) that perform actions on the build artifacts produced from the CI build for a release.

I think doing this for the Windows installer should be the first piece of this. In the medium term, I imagine this becoming something like a "post release script" which will be run after the CI build for the release has finished. For this specific issue, the actions we will need to perform locally are:

1. Download the Windows installer
2. Sign it
3. Upload the signed Windows installer so it can be added to a GitHub release.

When uploading the Windows installer, I think the two places we could reasonably upload it are:

1. Back to CI and let it deploy it to GH
2. To GH directly

I looked into 1 a bit and while Azure does allow you to pause a build and wait for manual interaction, I couldn't find a way to directly upload a file back to Azure. Instead, the best I think we could do is maintain an external server, upload the signed installer there, and have Azure download it from there when resuming the build. This seems far more complicated to me than just creating the GH release directly in the script running locally, but if someone has a clever idea here, I'm happy to reconsider.

Absent that though, for now I think we should write a script that we can run locally to create the GH release. Ideally, I think this script should help find and download the build artifacts automatically but this could potentially be added later if we want. If we want to add some helpful automation there now, Microsoft provides an [Azure](https://github.com/microsoft/azure-devops-python-api) Python package we could use.

Some of `cryptography`'s old release script can maybe used for inspiration: https://github.com/pyca/cryptography/blob/4f7715b6e6e6d9ee8a41e2e3615d5187d64ae013/release.py
Do you know why that's `cryptography`'s old release script? What did they switch to?
Their [current release script](https://github.com/pyca/cryptography/blob/master/release.py) is quite similar but they switched to GitHub Actions.

(If you're curious why they did that, between their GitHub and IRC I couldn't find any real explanation, however, they usually are on top of the latest trends and using GitHub Actions for CI is what all the cool kids seem to be doing nowadays. If you're curious why we went with Azure over GitHub Actions, I'll say:

1. They're quite similar with GitHub Actions being built on Azure.
2. You can read Adrien and I discussing it at length in the first comments of https://github.com/certbot/certbot/pull/8098 if you want.)

To be clear, [their release process](https://cryptography.io/en/latest/doing-a-release/) and what that script does is pretty different from ours, but I added the link to their script thinking it may be useful as an example of downloading artifacts from Azure using the library I linked above.
Wait a second, do we actually currently sign the windows release? That's not mentioned in the release process doc...
We do not but should in the next few months. That's blocked on us generating and distributing a new private key and obtaining a code signing certificate for that key.

Resolving patch:
diff --git a/certbot/setup.py b/certbot/setup.py
--- a/certbot/setup.py
+++ b/certbot/setup.py
@@ -93,8 +93,10 @@ def read_file(filename, encoding='utf8'):
 
 dev3_extras = [
     'astroid',
+    'azure-devops',
     'ipdb',
     'mypy',
+    'PyGithub',
     'pylint',
 ]
 
diff --git a/tools/create_github_release.py b/tools/create_github_release.py
new file mode 100755
--- /dev/null
+++ b/tools/create_github_release.py
@@ -0,0 +1,88 @@
+#!/usr/bin/env python
+"""
+Post-release script to download artifacts from azure pipelines and use them to create
+a GitHub release.
+
+Setup:
+ - Create a github personal access token
+   - https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token#creating-a-token
+   - You'll need repo scope
+   - Save the token to somewhere like ~/.ssh/githubpat.txt
+
+Run:
+
+python tools/create_github_release.py ~/.ssh/githubpat.txt
+"""
+
+import requests
+import sys
+import tempfile
+from zipfile import ZipFile
+
+from azure.devops.connection import Connection
+from github import Github
+
+def download_azure_artifacts(tempdir):
+    """Download and unzip build artifacts from Azure pipelines.
+
+    :param str path: path to a temporary directory to save the files
+
+    :returns: released certbot version number as a prefix-free string
+    :rtype: str
+
+    """
+    # Create a connection to the azure org
+    organization_url = 'https://dev.azure.com/certbot'
+    connection = Connection(base_url=organization_url)
+
+    # Find the build artifacts
+    build_client = connection.clients.get_build_client()
+    get_builds_response = build_client.get_builds('certbot', definitions='3')
+    build_id = get_builds_response.value[0].id
+    artifacts = build_client.get_artifacts('certbot', build_id)
+
+    # Save and unzip files
+    for filename in ('windows-installer', 'changelog'):
+        url = build_client.get_artifact('certbot', build_id, filename).resource.download_url
+        r = requests.get(url)
+        r.raise_for_status()
+        with open(tempdir + '/' + filename + '.zip', 'wb') as f:
+            f.write(r.content)
+        with ZipFile(tempdir + '/' + filename + '.zip', 'r') as zipObj:
+           zipObj.extractall(tempdir)
+
+    version = build_client.get_build('certbot', build_id).source_branch.split('v')[1]
+    return version
+
+def create_github_release(github_access_token, tempdir, version):
+    """Use build artifacts to create a github release, including uploading additional assets
+
+    :param str github_access_token: string containing github access token
+    :param str path: path to a temporary directory where azure artifacts are located
+    :param str version: Certbot version number, e.g. 1.7.0
+
+    """
+    # Create release
+    g = Github(github_access_token)
+    repo = g.get_user('certbot').get_repo('certbot')
+    release_notes = open(tempdir + '/changelog/release_notes.md', 'r').read()
+    release= repo.create_git_release('v{0}'.format(version),
+                                     'Certbot {0}'.format(version),
+                                     release_notes,
+                                     draft=True)
+
+    # Upload windows installer to release
+    release.upload_asset(tempdir + '/windows-installer/certbot-beta-installer-win32.exe')
+    release.update_release(release.title, release.body, draft=False)
+
+def main(args):
+    github_access_token_file = args[0]
+
+    github_access_token = open(github_access_token_file, 'r').read().rstrip()
+
+    with tempfile.TemporaryDirectory() as tempdir:
+        version = download_azure_artifacts(tempdir)
+        create_github_release(github_access_token, tempdir, version)
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/windows-installer/construct.py b/windows-installer/construct.py
--- a/windows-installer/construct.py
+++ b/windows-installer/construct.py
@@ -123,6 +123,8 @@ def _generate_pynsist_config(repo_path, build_path):
     certbot_version = subprocess.check_output([sys.executable, '-c', 'import certbot; print(certbot.__version__)'],
                                               universal_newlines=True, cwd=certbot_pkg_path).strip()
 
+    # If we change the installer name from `certbot-beta-installer-win32.exe`, it should
+    # also be changed in tools/create_github_release.py
     with open(installer_cfg_path, 'w') as file_h:
         file_h.write('''\
 [Application]


Test patch: