Add functionality in status_code_util to grab location from responses with 3xx errors. crawler_control uses this to make new url, but current maxdepthsearch disallows following the new url
Improve regex for urlinfo, less 404 errors
improve data structures, and their memory management
Add functions for search caching; not implemented in crawler_control.
complete rewrite of url caching
Improve urlinfo regex and give better output on links found from search engine
Merge branch 'combine_features'
Add caching functions, improved url parsing, and support for url redirection