Added multi-threading to crawler and updated handling of relative urls using folder references instead of file references. (Eg. linking to ‘categories’ instead of ‘categories/‘ caused the next level of link to remove ‘categories’ from the path)