Replace explict server shutdowns with unrefs