Updates to sphinx extensions

I split this off of #1355 to make it smaller. See that PR for full list of changes. Unfortunately, it's a bit difficult to split this one up any more, since they're a bit interdependent.
Actually, can you replace this unicode character with the html reference `&para;` please. This will neatly cut out all the unicode issues when building the docs and prevent any similar problems in end-user web browsers.

As a part of this change, can you also remove the encoding string from the top of the document.