Actually configure supervisor somewhere
This has been forgotten in 9809052db0b08f0f85368e889e2185419cc062a2