init
change file link with file
add requirements list
update copyright
Merge branch 'master' of https://bitbucket.org/veelion/xcrawler
move span_threshold as parameter of pop()
move span_threshold to class init() and renamed as span_of_host
move span_threshold to class init() and renamed as span_of_host
add worker_conf_file parameter to init()
fix: url_normalize() mistakes gbk to utf8 by _clean()
use traceback to print exception
update description of xcrawler
merge