decide to turn to some more advanced APIs, e.g. requests, beautifulsoup, etc
[Refactory] remove urllib2, cookieJar sort of thing, use requests instead
User info extraction done. MUCH easy than regex match once familiar with the BeautifulSoup APIs! They're really beautiful soup! Good night ;)
Now it can get separate fragments of the microposts pages and make up a full microposts page
[Fix] filter topics from following list
decide to write uids visited and uids queued into two separate files, and write a simple csv generator to avoid illusory memory overflow ;)
remove old regex stuff from weibo_crawler.py
Write uidsQueued & uidsVIsited in GetUidsThread into file on exit program
[Refactory] (GetUidsThread) change uidsQueued(Queue.Queue) into uidsVisited(set), because join on the queue is useless. The production rate is much higher than the consumption, and I cannot depend on this queue to end my program
Now it works fine :)