finish basic crawl and database input