Fix possibility of non-removed symlinked directory.

Fixes #4080
this should be swapped as far as I see. check for `is_dir` -> recurse, else unlink.