added a not
Refined regex, changed user agent, refactored out scraping into a function.