Package: crawl-me
crawl-me中文README.Crawl-me is a light-weight fast plugin based web picture crawler. You
can download your favorite pictures via the plugin if the website is
supported. For now, the plugins include gamersky and pixiv. If you want
to contribute, please just feel free to contact with me.Fork me on Github :)https://github.com/nyankosama/crawl-meFeaturesCrawl-me core supports muti-thread downloading using http
range-headers, so it’s very fast.It’s plugin based, so you can free add any plugin you want.Available pluginspixiv : This plugin allows you to download any author’s paintings inpixivsite.gamersky : This plugin supports downloading all pictures in special
topic fromgamerskysite.Installationinstall via pipMake sure you have already installedpython2.7andpip.Due to the fact that package relies on lxml, if your platform is linux,
please make sure you have installed lib libxslt-devel libxml2-devel. And
for windows please select a suitablelxml
installerto
install.And then:$ pip install crawl-meFor windows, please add {$python-home}/Scripts/ to systempathinstall via git1. UbuntuInstall the prerequisite library first:sudo apt-get install libxml2-dev
sudo apt-get install libxslt1-devAnd then you should installsetuptoolsin
order to run the setup.py filesudo apt-get install python-setuptoolsFinally, git clone the source, and install:$ git clone https://github.com/nyankosama/crawl-me.git
$ cd crawl-me/
$ sudo python setup.py install2. WindowsMake sure you have already installedpython2.7andpipYou can install python2.7 via windows installer. You can install pip via
downloading theget-pip.py,
and run it via python:python get-pip.pyAnd then install the prerequisite library lxml. please select a suitablelxml installerto
install.Finally git clone the source, and install:$ git clone https://github.com/nyankosama/crawl-me.git
$ cd crawl-me/
$ sudo python setup.py installFor windows, please add {$python-home}/Scripts/ to systempathUsageExamplesDownload 10 pages pictures at the url ofhttp://www.gamersky.com/ent/201404/352055.shtmlin gamersky site, and
store the pictures into local direcotry.crawl-me gamersky http://www.gamersky.com/ent/201404/352055.shtml ./gamersky-crawl 1 10Download all the paintings of 藤原(Fujiwara, Pixiv ID=27517), and
store them into local directory.crawl-me pixiv 27517 ./pixiv-crawl <your pixiv loginid> <your password>Command line optionsgeneral help$ crawl-me -h

usage: crawl-me [-h] plugin

positional arguments:
    plugin      plugin the crawler uses

optional arguments:
    -h, --help  show this help message and exit

available plugins:
----gamersky
----pixivgamersky$ crawl-me gamersky -h

usage: crawl-me [-h] plugin url savePath beginPage endPage

positional arguments:
    plugin      plugin the crawler uses
    url         your url to crawl
    savePath    the path where the imgs ars saved
    beginPage   the page where we start crawling
    endPage     the page where we end crawling

optional arguments:
    -h, --help  show this help message and exitpixiv$ crawl-me pixiv -h

usage: crawl-me [-h] plugin authorId savePath pixivId password

positional arguments:
    plugin      plugin the crawler uses
    authorId    the author id you want to crawl
    savePath    the path where the imgs ars saved
    pixivId     your pixiv login id
    password    your pixiv login password

optional arguments:
    -h, --help  show this help message and exitTODOFunctions:support breakpoint resumePlugins:weiboqq zoneLicensesMITChangeLog0.1.8Date: 2014-06-15add English README0.1.8dev-20140615Date: 2014-06-15bug fix:-v –version option load project.json fail0.1.8dev-20140612Date: 2014-06-12add -v –version option for main runnable file to show the package
version0.1.7Date: 2014-06-11add the http range header support auto-check0.1.6Date: 2014-06-11bug fix: terminal without colour doesnt display syslog prefix0.1.5Date: 2014-06-11bug fix:pip install bug in windows platform0.1.5dev-20140611Date: 2014-06-11bug fix:pypi data_files0.1.4Date: 2014-06-11the latest release0.1.4dev-20140611Date: 2014-06-11modify the README.md. Now it is consistent with rst format to display
on pypi0.1.4dev-20140610Date: 2014-06-10add support for installing from pip0.1.4dev3Date: 2014-06-10bug fixfix the binary write problem in windows platform0.1.4dev2Date: 2014-06-10add setuptools install support0.1.4dev1Date: 2014-06-09bug fixrangedownloader:http range-headers may not be supported0.1.3Date: 2014-06-07do some refactoryadd conf dictionary0.1.2Date: 2014-06-06add pluginpixiv0.1.1Date: 2014-06-05add plugingamersky0.0.1Date: 2014-06-05init the project