Package: imagebot
This bot (image scraper) crawls a given url(s) and downloads all the images.FeaturesSupported platforms: Linux / Windows / Python 2.7.Maintains a database of all downloaded images to avoid duplicate downloads.Optionally, it can scrape only under a particular url, e.g. scrapinghttp://website.com/albums/newwith this option will only download from new album.Filters urls by regex.Filters images by minimum size.Scrapes through javascript popup links (limited support).Live monitor window for displaying images as they are scraped.Asynchronous i/o design using scrapy and twisted.Usagecrawl command:Scrape images:imagebot crawl http://website.com
imagebot crawl http://website.com,http://otherwebsite.comOptions for crawl command:-d, –domainsScrape images while allowing images to be downloaded from other domain(s) (add multiple domains with comma separated list). The domain in the start url(s) is(are) allowed by default.imagebot crawlhttp://website.com-dotherwebsite.com,anotherwebsite.com-is, –images-storeSpecify image store location. Default: ~/Pictures/crawled/[jobname]imagebot crawlhttp://website.com-is/home/images-s, –min-sizeSpecify minimum size of image to be downloaded (width x height).imagebot crawlhttp://website.com-s300x300-u, –stay-underStay under the start url. Only those urls that have the start url as prefix will be crawled. Useful, for example, to crawl an album or a subsection on a website.imagebot crawlhttp://website.com/albums/new-u-m, –monitorLaunch monitor window for displaying images as they are scraped.imagebot crawlhttp://website.com-m-a, –user-agentSet user-agent string. Default: imagebot. It is recommended to change it to identify your bot as a matter of responsible crawling.imagebot crawlhttp://website.com-a"my_imagebot(http://mysite.com)"-r, –url-regexSpecify regex for urls. Only those urls matching the regex will be crawled. It does not apply to start url(s).imagebot crawlhttp://website.com-r.*gallery.*-dl, –depth-limitSpecify depth limit for crawling. Use value of 0 to scrape only on start url(s).imagebot crawlhttp://website.com-dl2–no-cdnsA list of well known cdn’s is included and enabled by default for image downloads. Use this option to disable it.-at, –auto-throttleEnable auto throttle feature of scrapy. (details in scrapy docs).-j, –jobnameSpecify a job name. This will be used to store image meta data in the database. By default, domain name of the start url is used as the job name.-nc, –no-cacheDisable http caching.-l, –log-levelSpecify log level.
Supported levels: info, silent, critical, error, debug, warning. Default: error.imagebot crawlhttp://website.com-ldebug-h, –helpGet help on crawl command options.clear command:This command is useful for various kinds of cleanup.Options for clear command:–cacheClear http cache.–dbRemove image metadata for a job from the database.imagebot clear--dbwebsite.com–duplicate-imagesMultiple copies of same image may be downloaded due to different urls. Use this option to delete duplicate images for a job.imagebot clear--duplicate-imageswebsite.com-h, –helpGet help on clear command options.Dependenciespywin32 (http://sourceforge.net/projects/pywin32/)Needed on Windows.python-gi (Python GObject Introspection API)Needed on Linux for gtk UI. (Optional). If not found, python built-in Tkinter will be used.
On Ubuntu:apt-getinstallpython-giscrapy (web crawling framework)It will be automatically installed by pip.Pillow (Python Imaging Library)It will be automatically installed by pip.DownloadPyPI:http://pypi.python.org/pypi/imagebot/Source:https://github.com/amol9/imagebot/