Don't delete empty directories without recurse