Only log about removing a server when actually removed