add the work done to crawl and clean the datasets