this scrips pulls the scrape urls from the DB, then identifes all
of the links on the page that are not PDF's.  This is used as a
pre-processor to speed up the actual PDF scrapers