added code to walk data directories
updating crawl.R