wrapped crawl with an http semaphore to limit http.Get calls