Close down channels that we open in RpcServer