Remove keepalived.conf as well