Package: gerapy-proxy
Gerapy ProxyThis is a package for supporting proxy with async mechanism in Scrapy, also this
package is a module inGerapy.Installationpip3installgerapy-proxyUsageIf you have a ProxyPool which can provide a random proxy for every request, you can use this package
to integrate proxy into your Scrapy/Gerapy Project.For example, there is aProxyPool APIwhich can return a random proxy
per time, we can configureGERAPY_PROXY_POOL_URLsetting provided by this package to enable proxy for every Scrapy Request.To use this package, firstly install it and then enable it in DownloadMiddleware:DOWNLOADER_MIDDLEWARES={'gerapy_proxy.middlewares.ProxyPoolMiddleware':543,}and add proxy url in settings:GERAPY_PROXY_POOL_URL='https://proxypool.scrape.center/random'This ProxyPool is configured based on thisProxyPoolrepo, you can
also build your own ProxyPool service.Now, you've finished it.TheProxyPoolMiddlewarewill firstly fetch a proxy fromGERAPY_PROXY_POOL_URLand setmeta.proxyattribute
to Scrapy Reqeust.ConfigurationBasic AuthIf your ProxyPool has Basic Auth, you can enable it by configuring these settings:GERAPY_PROXY_POOL_AUTH=TrueGERAPY_PROXY_POOL_USERNAME=<username>GERAPY_PROXY_POOL_PASSWORD=<password>Min Retry TimesIf you want to enable Proxy depends on the retry times, you can configure this settings:GERAPY_PROXY_POOL_MIN_RETRY_TIMES=2Then proxy will only work if the retry times of Request greater or equal than 2.Random EnabledIf you want to enable the proxy randomly, you can configure the probability of enabling it:GERAPY_PROXY_POOL_RANDOM_ENABLE_RATE=0.8Then probability of enabling the proxy is 80%, if you configure it to 1, proxy will always be enabled.Fetch TimeoutYou can also configure the max time of fetching proxy from ProxyPool:GERAPY_PROXY_POOL_TIMEOUT=5After configuring this, if Proxy Pool does not return result in 5s, proxy will not be used.ProxyPool Response ParserYour ProxyPool may not return the same format asthisin plain text,
you can also define a parser to extract proxy from your ProxyPool.For example, if your ProxyPool return this for every request:{"host":"111.222.223.224","port":3128}You can define a method like:importjsondefparse_result(text):data=json.loads(text)returnf'{data.get("host")}:{data.get("port")}'GERAPY_PROXY_EXTRACT_FUNC=parse_resultThen you will get the proxy with correct format.ExampleFor more detail, please seeexample.Also you can directly run with Docker:docker run germey/gerapy-proxy-exampleOutputs:2020-07-1519:17:34[scrapy.utils.log]INFO:Scrapy2.2.0started(bot:example)2020-07-1519:17:34[scrapy.utils.log]INFO:Versions:lxml4.3.3.0,libxml22.9.9,cssselect1.1.0,parsel1.6.0,w3lib1.22.0,Twisted20.3.0,Python3.7.7(default,May62020,04:59:01)-[Clang4.0.1(tags/RELEASE_401/final)],pyOpenSSL19.1.0(OpenSSL1.1.1d10Sep2019),cryptography2.8,PlatformDarwin-19.4.0-x86_64-i386-64bit2020-07-1519:17:34[scrapy.utils.log]DEBUG:Usingreactor:twisted.internet.selectreactor.SelectReactor2020-07-1519:17:34[scrapy.crawler]INFO:Overriddensettings:{'BOT_NAME':'example','CONCURRENT_REQUESTS':3,'DOWNLOAD_TIMEOUT':10,'NEWSPIDER_MODULE':'example.spiders','RETRY_TIMES':10,'SPIDER_MODULES':['example.spiders']}2020-07-1519:17:34[scrapy.extensions.telnet]INFO:TelnetPassword:33299ca0ce64f2152020-07-1519:17:34[scrapy.middleware]INFO:Enabledextensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.logstats.LogStats']2020-07-1519:17:34[asyncio]DEBUG:Usingselector:KqueueSelector2020-07-1519:17:34[scrapy.middleware]INFO:Enableddownloadermiddlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','gerapy_proxy.middlewares.ProxyPoolMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']2020-07-1519:17:34[scrapy.middleware]INFO:Enabledspidermiddlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2020-07-1519:17:34[scrapy.middleware]INFO:Enableditempipelines:[]2020-07-1519:17:34[scrapy.core.engine]INFO:Spideropened2020-07-1519:17:34[scrapy.extensions.logstats]INFO:Crawled0pages(at0pages/min),scraped0items(at0items/min)2020-07-1519:17:34[scrapy.extensions.telnet]INFO:Telnetconsolelisteningon127.0.0.1:60232020-07-1519:17:34[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:34[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:getproxy113.124.94.189:99992020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:getproxy84.53.238.49:235002020-07-1519:17:35[gerapy_proxy.middlewares]DEBUG:getproxy217.150.77.31:532812020-07-1519:17:40[scrapy.core.engine]DEBUG:Crawled(200)<POSThttps://httpbin.org/delay/3>(referer:None)2020-07-1519:17:40[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:40[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:40[example.spiders.httpbin]INFO:gotrequestfrom113.124.94.189successfully,currentpage12020-07-1519:17:40[gerapy_proxy.middlewares]DEBUG:getproxy144.52.244.3:99992020-07-1519:17:45[scrapy.downloadermiddlewares.retry]DEBUG:Retrying<POSThttps://httpbin.org/delay/3>(failed1times):Usertimeoutcausedconnectionfailure:Gettinghttps://httpbin.org/delay/3tooklongerthan10.0seconds..2020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:45[scrapy.downloadermiddlewares.retry]DEBUG:Retrying<POSThttps://httpbin.org/delay/3>(failed1times):Usertimeoutcausedconnectionfailure:Gettinghttps://httpbin.org/delay/3tooklongerthan10.0seconds..2020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:starttogetproxyfromproxypool2020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:getproxyusingkwargs{'timeout':5,'url':'https://proxypool.scrape.center/random'}2020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:getproxy1.20.101.149:447782020-07-1519:17:45[gerapy_proxy.middlewares]DEBUG:getproxy105.27.116.46:56792