sumityadav6541/unigo — pull request #1 by shreyasmita (2014-07-06 10:01:51+00:00)
Title: add top domains

Top500domains.csv: contains all website urls sorted as per their google page rank( 10 to 5)

link_extractor.py: just extracts links from .csv file and returns it to crawler.py

crawler_try.py: imports link_extractor.py and open each url using urllib2... This is intended to download or fetch relevant images to our no sql db (yet to be done)