Large site scanner engine

For those extra hard cases where a directory contains hundreds or thousands of files