seperate crawling and parsing
index for done
DoneSet in memory