[core] improve OS X docs build information.

Homebrew's graphviz now installs dot properly, and we want
sphinx & docutils (the former isn't currently used by -core, but
we only have one HACKING doc for the whole system).

Also install doxygen and pngcrush, which allows a full
maintainer build to work under OS X.