Move crawling functionality to external library