initial commit
updated deprecated setting
removed no longer working google diretory spider
add DirbotItem to items.py so that genspider creates a valid spider
Merge pull request #1 from stav/master

`scrapy genspider` creates `import DirbotItem` but I get ImportError: cannot import name DirbotItem
removed references to googledir spider from README. closes #2
remove deprecated settings usage
add setup.py to easy install on venvs
no need to have two item types
added spider contract to dmoz spider
Fix xpathselector. Some unwanted entries are also selected if a class is not specified on ul.
Merge pull request #3 from akshar-raaj/master

Added a class on selector
capture description without trailing newline characters using a regular expression
remove custom __str__ from Website item
Removed reference to ``crawl URL`` command as it's no longer supported.
Merge pull request #5 from darkrho/readme-update

Removed reference to ``crawl URL`` command as it's no longer supported.
Merge branch 'dirbot'