Fix for leftover empty directories during package removal

Fixes #1675 second part