Initial implementation of the directory harvester for indexing, and the search API with weightings and limits.