wrapped a bunch of code in Try's in case of urls not being allowed for crawling