urls to crawl are now stored in a single hash, rather than multiple hashes associated with their parent urls
Added back original check for end of pdftribute.net urls since they seemed to change it back on their end