Avoid unnecessary calls to wildignore

This is significantly faster on large file lists