Replace explict server shutdowns with unrefs and add port fallback