Added condition to do a GET if link is to be crawled, else to do a HEAD, then a GET if HEAD fails
Added counter for urls checked so far. Updated logging