Avoid Readdir() call for non-pattern parts

Avoids recursion with certain rulefs setups.