adding gunicorn, procfile
adding web interface for crawler
adding test for pipelines
writing outlinks to separate file using offsitemiddlware
Processing links to remove mulitple '/'
Adding core app, test method for sspider rich files
Removing todos
Removing old settings file
Some html testing files
Adding deny domains for libraries and updating project requirements
Removing the '__author__' variable in *.py files,
Changing spider server from scrapy to scrapyd
Adding new model and data to handle urls to avoid
Improving use of scrapyd, adding communicator, correcting scrapyd launch folder
ScrapydCommunicator working :-), able to start crawls via web interface
Adding more exceptions to gitignore
Synchronizing name of avoidurl in models and fixtures
Adding south
Adding base requirement file, modifying others to import it
Changing import of multiple items to use brackets
Separating communicator from johnnywalker's view to core
Adding migrations for apps
Changing tests to use core.test.Testcase, adding primitive stats generator for domains and its tests
Adding coverage and changing test db from postgresql to in-memory sqlite3