Remove package directories without warnings when they only contain empty dirs

Closes #1740 (not exactly what was asked for, but this seems alright)