a slightly better crawler
refactor crawler