show more info in the urls csv file.
record whether we processed the url or not
in the SpiderReply. normalize /index.html etc.
to / for the outlinks. in Links.cpp class.