initial commit
updated deprecated setting
removed no longer working google diretory spider
add DirbotItem to items.py so that genspider creates a valid spider
Merge pull request #1 from stav/master

`scrapy genspider` creates `import DirbotItem` but I get ImportError: cannot import name DirbotItem
removed references to googledir spider from README. closes #2
remove deprecated settings usage
add setup.py to easy install on venvs
no need to have two item types
added spider contract to dmoz spider
Fix xpathselector. Some unwanted entries are also selected if a class is not specified on ul.
Merge pull request #3 from akshar-raaj/master

Added a class on selector
capture description without trailing newline characters using a regular expression
remove custom __str__ from Website item
修改配置
页面收集完成
修改
添加xpath
更名
scrapy 策略修改完成