support new format and add crawling