Remove empty directories only if they match exclude pattern.

Current exclude logic even removes the directories left empty after "exclude match" but which 
don't match the exclude pattern.

This for example makes empty directories maintained under subversion 
to be excluded (via `--exclude '**.svn*'`) even though intention is to include them.