Updates to sphinx extensions

I split this off of #1355 to make it smaller. See that PR for full list of changes. Unfortunately, it's a bit difficult to split this one up any more, since they're a bit interdependent.
Being of a particularly picky nature, I spotted that in this file:
 * `%` is used for string formatting rather than the now preferred `str.format()`,
 * strings are mostly defined with single quotes, but then suddenly double quotes start getting used, and
 * comments should be proper English sentences.