Added .gitignore
Functioning crawler
Added requirements.txt and rebuilt the system entirely
Merge branch 'master' of https://github.com/nkartashov/NKrawler