Added extension point to use custom crawling method rather than the
default crawling method. This allows one to pass authentication, cookies
etc in a better way.