Also remove unused directories.