Make sure supervisor reloads.

If we change the config I presume we need to reload the
thing in supervisor.