Package: rentier-scrapy-proxy-rotator
scrapy-rotating-proxiesThis package provides aScrapymiddleware to use rotating proxies,
check that they are alive and adjust crawling speed.License is MIT.Installationpip install scrapy-rotating-proxiesUsageAddROTATING_PROXY_LISToption with a list of proxies to settings.py:ROTATING_PROXY_LIST = [
    'proxy1.com:8000',
    'proxy2.com:8031',
    # ...
]As an alternative, you can specify aROTATING_PROXY_LIST_PATHoptions
with a path to a file with proxies, one per line:ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt'ROTATING_PROXY_LIST_PATHtakes precedence overROTATING_PROXY_LISTif both options are present.Then add rotating_proxies middlewares to your DOWNLOADER_MIDDLEWARES:DOWNLOADER_MIDDLEWARES = {
    # ...
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
    # ...
}After this all requests will be proxied using one of the proxies from
theROTATING_PROXY_LIST/ROTATING_PROXY_LIST_PATH.Requests with “proxy” set in their meta are not handled by
scrapy-rotating-proxies. To disable proxying for a request setrequest.meta['proxy']= None; to set proxy explicitly userequest.meta['proxy']="<my-proxy-address>".ConcurrencyBy default, all default Scrapy concurrency options (DOWNLOAD_DELAY,AUTHTHROTTLE_...,CONCURRENT_REQUESTS_PER_DOMAIN, etc) become
per-proxy for proxied requests when RotatingProxyMiddleware is enabled.
For example, if you setCONCURRENT_REQUESTS_PER_DOMAIN=2then
spider will be making at most 2 concurrent connections to each proxy,
regardless of request url domain.Customizationscrapy-rotating-proxieskeeps track of working and non-working proxies,
and re-checks non-working from time to time.Detection of a non-working proxy is site-specific.
By default,scrapy-rotating-proxiesuses a simple heuristic:
if a response status code is not 200, response body is empty or if
there was an exception then proxy is considered dead.You can override ban detection method by passing a path to
a custom BanDectionPolicy inROTATING_PROXY_BAN_POLICYoption, e.g.:# settings.py
ROTATING_PROXY_BAN_POLICY = 'myproject.policy.MyBanPolicy'The policy must be a class withresponse_is_banandexception_is_banmethods. These methods can return True
(ban detected), False (not a ban) or None (unknown). It can be convenient
to subclass and modify default BanDetectionPolicy:# myproject/policy.py
from rotating_proxies.policy import BanDetectionPolicy

class MyPolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        # use default rules, but also consider HTTP 200 responses
        # a ban if there is 'captcha' word in response body.
        ban = super(MyPolicy, self).response_is_ban(request, response)
        ban = ban or b'captcha' in response.body
        return ban

    def exception_is_ban(self, request, exception):
        # override method completely: don't take exceptions in account
        return NoneInstead of creating a policy you can also implementresponse_is_banandexception_is_banmethods as spider methods, for example:class MySpider(scrapy.Spider):
    # ...

    def response_is_ban(self, request, response):
        return b'banned' in response.body

    def exception_is_ban(self, request, exception):
        return NoneIt is important to have these rules correct because action for a failed
request and a bad proxy should be different: if it is a proxy to blame
it makes sense to retry the request with a different proxy.Non-working proxies could become alive again after some time.scrapy-rotating-proxiesuses a randomized exponential backoff for these
checks - first check happens soon, if it still fails then next check is
delayed further, etc. UseROTATING_PROXY_BACKOFF_BASEto adjust the
initial delay (by default it is random, from 0 to 5 minutes). The randomized
exponential backoff is capped byROTATING_PROXY_BACKOFF_CAP.SettingsROTATING_PROXY_LIST- a list of proxies to choose from;ROTATING_PROXY_LIST_PATH- path to a file with a list of proxies;ROTATING_PROXY_LOGSTATS_INTERVAL- stats logging interval in seconds,
30 by default;ROTATING_PROXY_CLOSE_SPIDER- When True, spider is stopped if
there are no alive proxies. If False (default), then when there is no
alive proxies all dead proxies are re-checked.ROTATING_PROXY_PAGE_RETRY_TIMES- a number of times to retry
downloading a page using a different proxy. After this amount of retries
failure is considered a page failure, not a proxy failure.
Think of it this way: every improperly detected ban cost youROTATING_PROXY_PAGE_RETRY_TIMESalive proxies. Default: 5.It is possible to change this option per-request usingmax_proxies_to_tryrequest.meta key - for example, you can use a higher
value for certain pages if you’re sure they should work.ROTATING_PROXY_BACKOFF_BASE- base backoff time, in seconds.
Default is 300 (i.e. 5 min).ROTATING_PROXY_BACKOFF_CAP- backoff time cap, in seconds.
Default is 3600 (i.e. 60 min).ROTATING_PROXY_BAN_POLICY- path to a ban detection policy.
Default is'rotating_proxies.policy.BanDetectionPolicy'.FAQQ: Where to get proxy lists? How to write and maintain ban rules?A: It is up to you to find proxies and maintain proper ban rules
for web sites;scrapy-rotating-proxiesdoesn’t have anything built-in.
There are commercial proxy services likehttps://crawlera.com/which can
integrate with Scrapy (seehttps://github.com/scrapy-plugins/scrapy-crawlera)
and take care of all these details.Contributingsource code:https://github.com/TeamHG-Memex/scrapy-rotating-proxiesbug tracker:https://github.com/TeamHG-Memex/scrapy-rotating-proxies/issuesTo run tests, installtoxand runtoxfrom the source checkout.CHANGES0.6 (2018-12-28)Proxy information is added to scrapy stats:proxies/uncheckedproxies/reanimatedproxies/deadproxies/goodproxies/mean_backoff0.5 (2017-10-09)ROTATING_PROXY_LIST_PATHoption allows to pass file name
with a proxy list.0.4 (2017-06-06)ROTATING_PROXY_BACKOFF_CAPoption allows to change max backoff time
from the default 1 hour.0.3.2 (2017-06-05)fixed proxy authentication issue.0.3.1 (2017-03-20)fixed OverflowError during backoff computation.0.3 (2017-03-14)redirects with empty bodies are no longer considered bans
(thanks Diga Widyaprana).ROTATING_PROXY_BAN_POLICYoption allows to customize ban detection
for all spiders.0.2.3 (2017-03-03)max_proxies_to_tryrequest.meta key allows to overrideROTATING_PROXY_PAGE_RETRY_TIMESoption per-request.0.2.2 (2017-03-01)Update default ban detection rules: scrapy.exceptions.IgnoreRequest
is not a ban.0.2.1 (2017-02-08)changedROTATING_PROXY_PAGE_RETRY_TIMESdefault value - it is now 5.0.2 (2017-02-07)improved default ban detection rules;log ban stats.0.1 (2017-02-01)Initial release