removing extra file
Merge pull request #2 from chernjie/master

Net::HTTP.new only accepts hostname, does not care about scheme
added spider configuration file
allow NTLM auth
Minor changes

Conflicts:
	spider.cfg
modified spider config file

Conflicts:
	spider.cfg
v0.2