Basic spider to index all files and pages on a given website and to store key values.