Yes, you want to just `rm` links.   But `isdir` won't work: it returns `true` for a symlink to a directory.