Merge new crawler code.