Faster removal of RPC servers