Fix possibility of non-removed symlinked directory.

Fixes #4080
Why do you think so? Do you mean `is_file()` may return `true` for the symlink on a directory?