Add rem_server function, along with keeping track of refs to a server