committing changes after website crawl