add Apache License 2.0
simplify the mongo writer
finish http://www.chinawutong.com/103.html?pid=2
finish http://wb.56888.net/OutSourceList.aspx?tendertype=4&p=1
finish http://www.51yunli.com/goods/0/0/1/0
abstract crawler.crawl()
abstract crawler.lifetime(), fixes #1
initial tornado
abstract the mongodb module
finish http://www.0256.cn/goods/?PageIndex=1
rewrite crawler_runner, now we could call the different crawler on different site
change the MongoDB.last() method, return a json list instead of a json item
Merge branch 'tornado'
change the MongoDB.last() method, return a json list instead of a json item
check the validity of data by adding /Server/mongo/maintenance.py
fix #4
finish http://www.8glw.com/main_info.asp?id=1&page=1
Merge branch 'mongo'
add MongoDB.find_one() and replace the Crawler.find() with Crawler.exist()
kill the dead crawler every 4 hours
init nginx and supervisor