Remove symlinks instead of recursing into them when deleting, fixes #3050