moved routines into functions, expanded pattern search, re-check failed urls, additional sanity checks