xbps-src: remove_pkg: exit early if destdir does not exist.