Support remote connections.

This changes the broker start to be lazy, since we don't want to ever
start if all we use is remote connections.