Only reload when things change if we are not connected

If we are connected we will be informed over the socket, so don't
need to do it manually.

Tests
--------
Test that we now only reload once on a change.