updating new formates
spiders updated
unused field commented
link to server is updated
http cache is disabled
a general spider which takes seed url and domain as arguments