scrapy/scrapely — pull request #45 by xyb (2013-09-10 05:07:11+00:00)
Title: support CJK string annotation; print readably CJK string in scrapely.tool's output

scrapely.tool will crash when using CJK string as annotation in scrapely.tool:

```
$ python -m scrapely.tool blog.json
scrapely> ta http://blog.douban.com/douban/2013/07/04/2630/
[0] http://blog.douban.com/douban/2013/07/04/2630/
scrapely> t 0 算法工程师如何改进豆瓣电影 TOP250
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Users/xyb/scrapely.xyb/scrapely/tool.py", line 189, in
main()
File "/Users/xyb/scrapely.xyb/scrapely/tool.py", line 186, in main
t.cmdloop()
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cmd.py", line 142, in cmdloop
stop = self.onecmd(line)
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cmd.py", line 221, in onecmd
return func(arg)
File "/Users/xyb/scrapely.xyb/scrapely/tool.py", line 48, in do_t
selection = apply_criteria(criteria, tm)
File "/Users/xyb/scrapely.xyb/scrapely/tool.py", line 147, in apply_criteria
sel = tm.select(func)
File "scrapely/template.py", line 48, in select
score = score_func(fragment, htmlpage)
File "scrapely/template.py", line 95, in func
if text in fdata:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 0: ordinal not in range(128)
```

I fixed it, and add improved the usability of scrapely.tool's output that including CJK unicode characters:

```
$ python -m scrapely.tool blog.json
scrapely> t 0 算法工程师如何改进豆瓣电影 TOP250
[0] u'

算法工程师如何改进豆瓣电影 TOP250

'
[1] u'

豆瓣blog » Blog Archive » 算法工程师如何改进豆瓣电影 TOP250

'
[2] u''
scrapely>
```