Added an option to allow DirectoryLocator to be non-recursive
Updated docs.
Made URL scoring a method for use by subclasses if desired.