Remove directories when removing an item.
Towards #295