Fix server destroy throwing when not started