Remove currently empty directories