konklone/konklone — pull request #126 by konklone (2014-04-06 23:56:16+00:00)
Title: Sync posts FROM GitHub

After adding syncing **to** GitHub in #125, where updates to posts inside my blog's CMS get synced to GitHub, this PR adds support for the other direction.

Any repo can configure a webhook that pushes to `/github/sync`. That endpoint will nicely print out `ping` events, process `push` events, and reject all other types with a `400` response code.

The endpoint will accept pushes to any file whose URL has been linked to a post. By using a URL, this locks updates per-post to a particular repository, branch, and path. The pushes can come from multiple repositories, as long as there's a matching URL.

The sync endpoint authenticates using a shared secret, which the GitHub API allows you to set on a webhook via their API. This shared secret is verified using the [same algorithm GitHub uses](https://github.com/github/github-services/blob/f3bb3dd780feb6318c42b2db064ed6d481b70a1f/lib/service/http_helper.rb#L77).

There are two safeguards to prevent infinite sync loops. When a post is saved in the CMS, and it correctly syncs content over to GitHub, the resulting commit SHA is saved in an ongoing list of seen commit SHAs, in the post's `github_commits` field. Because GitHub's webhooks always fire no matter what, GitHub will boomerang that update back, but the webhook endpoint checks incoming SHAs against the list of seen SHAs and ignores it.

When a post is updated in GitHub, and it correctly syncs content over to the blog via the webhook, the webhook sets the temporary Post attribute `needs_sync` to `false`, which suppresses the resulting database save from re-POST-ing over to GitHub.

In this way, sync should operate from a file in any repo I choose (as long as the webhook uses the same shared secret), with any blog post I choose to link that file to. Updates from non-chosen branches should be ignored, and loops prevented. I think it should Just Work.

Some bumps along the way:

* Webhook shared secrets can't be set in the browser UX, for some reason. All the other documented `config` options, like disabling SSL verification, can. So I had to do a one-time API operation (via [octokit](https://github.com/octokit/octokit.rb) in irb) to set that secret. I've filed a support ticket about the issue.
* The developer API documentation had inconsistent notation of what header the GitHub signature would arrive in, `X-Hub-Signature` and `X-GitHub-Signature`. The former is correct, and I submitted [a PR to fix this in the docs](https://github.com/github/developer.github.com/pull/475), which has been merged.
* There's no easy programmatic way to distinguish a merge commit from a normal commit. Right now, the only way I prevent a merge commit from causing a post to be re-updated is to compare the current and proposed post body and see if they're the same.
* Octokit makes you decode the base64 body of the content yourself, and it's not obvious that you also need to force the encoding to be UTF-8 yourself as well. Took some [SO-ing](http://stackoverflow.com/a/16596591/16075) to get to the bottom of it.
* How to process git data is not very intuitive -- there is a `commits` array and a `head_commit` object, there's a `distinct` boolean flag that means something - without a prior understanding of git internals, it took me a while to grok this.

But overall, this whole process was (seriously) an absolute joy. The webhook development/debugging UX and features are superb. The API is generally well-thought out and reliable. And even though there's a lot in this description, conceptually it's not that difficult of a system to manage and keep in my head.

Also, I love that GitHub is linking prominently to [ngrok](https://ngrok.com/), which has been my companion for a while (and maybe I should start paying for it).

To be done after this PR: updating the blog front-end to add a prominent link to contribute to a post, if that post is GitHub-linked.

/cc @parkr @benbalter pull requests to [blog posts](https://github.com/konklone/writing/tree/master/blog) welcome :)