When the server is deleted we first stop brokering.

This stops segfaults when a server goes out of scope.