Only close server if we opened it