depth_client: make robust to servers coming and going
server: fix race condition where server could go away before we query it