Package: scrapy-rotated-proxy
OverviewScrapy-Rotated-Proxy is a Scrapy downloadmiddleware to dynamically attach proxy to Request,
which can repeately use rotated proxies supplied by configuration.
It can temporarily block unavailable proxy ip
and retrieve to use in the future when the proxy is available.
Also, it can remove invalid proxy ip through Scrapy signal.
Proxy ip list can be supplied through Spider Settings, File or MongoDB.RequirementsPython 2.7 or Python 3.3+Works on Linux, Windows, Mac OSX, BSDInstallThe quick way:pip install scrapy-rotated-proxyOR copy this middleware to your scrapy project.ConfigurationBasic ConfigurationEnable with Spider Settingsenable scrapy-rotated-proxy middleware and supply proxy ip list through Spider Settings# -----------------------------------------------------------------------------
# ROTATED PROXY SETTINGS (Spider Settings Backend)
# -----------------------------------------------------------------------------
DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
    'scrapy_rotated_proxy.downloadmiddlewares.proxy.RotatedProxyMiddleware': 750,
})
ROTATED_PROXY_ENABLED = True
PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.file_storage.FileProxyStorage'
# When set PROXY_FILE_PATH='', scrapy-rotated-proxy
# will use proxy in Spider Settings default.
PROXY_FILE_PATH = ''
HTTP_PROXIES = [
    'http://proxy0:8888',
    'http://user:pass@proxy1:8888',
    'https://user:pass@proxy1:8888',
]
HTTPS_PROXIES = [
    'http://proxy0:8888',
    'http://user:pass@proxy1:8888',
    'https://user:pass@proxy1:8888',
]Enable with Local Fileenable scrapy-rotated-proxy middleware and supply proxy ip list through Local File# -----------------------------------------------------------------------------
# ROTATED PROXY SETTINGS (Local File Backend)
# -----------------------------------------------------------------------------
DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
    'scrapy_rotated_proxy.downloadmiddlewares.proxy.RotatedProxyMiddleware': 750,
})
ROTATED_PROXY_ENABLED = True
PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.file_storage.FileProxyStorage'
PROXY_FILE_PATH = 'file_path/proxy.txt'local file store proxy list with json style# proxy file content, must conform to json format, otherwise will cause json
# load error
HTTP_PROXIES = [
    'http://proxy0:8888',
    'http://user:pass@proxy1:8888',
    'https://user:pass@proxy1:8888'
]
HTTPS_PROXIES = [
    'http://proxy0:8888',
    'http://user:pass@proxy1:8888',
    'https://user:pass@proxy1:8888'
]Enable with MongoDBenablescrapy-rotated-proxymiddleware and supply proxy ip list through MongoDB# -----------------------------------------------------------------------------
# ROTATED PROXY SETTINGS (MongoDB Backend)
# -----------------------------------------------------------------------------
# mongodb document required field: scheme, ip, port, username, password
# document example: {'scheme': 'http', 'ip': '10.0.0.1', 'port': 8080,
# 'username':'user', 'password':'password'}
DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
    'scrapy_rotated_proxy.downloadmiddlewares.proxy.RotatedProxyMiddleware': 750,
})
ROTATED_PROXY_ENABLED = True
PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.mongodb_storage.MongoDBProxyStorage'
PROXY_MONGODB_HOST = HOST_OR_IP
PROXY_MONGODB_PORT = 27017
PROXY_MONGODB_USERNAME = USERNAME_OR_NONE
PROXY_MONGODB_PASSWORD = PASSWORD_OR_NONE
PROXY_MONGODB_AUTH_DB = 'admin'
PROXY_MONGODB_DB = 'vps_management'
PROXY_MONGODB_COLL = 'service'Advanced ConfigurationBlock SettingsDefault, spider will close when run out of all proxies. you can config spider to
wait until block proxies become valid, which you block by signal# -----------------------------------------------------------------------------
# OTHER SETTINGS (Optional)
# -----------------------------------------------------------------------------
PROXY_SLEEP_INTERVAL = 60*60*24  # Default 24hours
PROXY_SPIDER_CLOSE_WHEN_NO_PROXY = False # Default TrueSignalsRemove proxy that never be used in the spider, you can send signal toscrapy_rotated_proxy.signals.proxy_remove, which signal must contains arguments
includingspider,request,exceptionBlock proxy that can be used in the future after sleep interval reach, you can send signal toscrapy_rotated_proxy.signals.proxy_block, which signal must contains arguments
includingspider,response,exceptionSettings ReferenceSettingDescriptionDefaultROTATED_PROXY_ENABLEDWhether to enable Scrapy-Rotated-ProxyTruePROXY_STORAGEA class which implements the proxy storage backendFileProxyStoragePROXY_MONGODB_HOSTMongoDB host for MongoDB backend‘127.0.0.1’PROXY_MONGODB_PORTMongoDB port for MongoDB backend27017PROXY_MONGODB_USERNAMEMongoDB username for MongoDB backendNonePROXY_MOGNODB_PASSWORDMongoDB password for MongoDB backendNonePROXY_MONGODB_DBMongoDB database name for MongoDB backendproxy_managementPROXY_MONGODB_COLLMongoDB collection name for MongoDB backendproxyPROXY_MONGODB_OPTIONS_*MongoDB uri options for MongoDB backendPROXY_FILE_PATHPath of file that store proxies. default is None, means get proxies from Spider SettingsNoneHTTP_PROXIESkeywords of HTTP proxies for LocalFile backend or Spider SettingsHTTPS_PROXIESkeywords of HTTPS proxies for LocalFile backend or Spider SettingsPROXY_SLEEP_INTERVALTime to sleep for blocked proxy become available60*60*24PROXY_SPIDER_CLOSE_WHEN_NO_PROXYWhether to close spider when run out of all proxiesTruePROXY_RELOAD_ENABLEDenable to reload proxy from storage when all proxies was used and prepare to cycle useFalse