added combined crawl, merge function