bulk commit and add the crawler files