Package: scrapy-fake-useragent
scrapy-fake-useragentRandom User-Agent middleware for Scrapy scraping framework based onfake-useragent, which picks upUser-Agentstrings
based onusage statisticsfrom areal world database, but also has the option to configure a generator
of fake UA strings, as a backup, powered byFaker.It also has the possibility of extending the
capabilities of the middleware, by adding your own providers.ChangesPlease seeCHANGELOG.InstallationThe simplest way is to install it viapip:pip install scrapy-fake-useragentConfigurationTurn off the built-inUserAgentMiddlewareandRetryMiddlewareand addRandomUserAgentMiddlewareandRetryUserAgentMiddleware.In Scrapy >=1.0:DOWNLOADER_MIDDLEWARES={'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None,'scrapy.downloadermiddlewares.retry.RetryMiddleware':None,'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware':400,'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware':401,}In Scrapy <1.0:DOWNLOADER_MIDDLEWARES={'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware':None,'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware':None,'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware':400,'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware':401,}Recommended setting (1.3.0+):FAKEUSERAGENT_PROVIDERS=['scrapy_fake_useragent.providers.FakeUserAgentProvider',# this is the first provider we'll try'scrapy_fake_useragent.providers.FakerProvider',# if FakeUserAgentProvider fails, we'll use faker to generate a user-agent string for us'scrapy_fake_useragent.providers.FixedUserAgentProvider',# fall back to USER_AGENT value]USER_AGENT='<your user agent string which you will fall back to if all other providers fail>'Additional configuration informationEnabling providersThe package comes with a thin abstraction layer of User-Agent providers, which for purposes of backwards compatibility defaults to:FAKEUSERAGENT_PROVIDERS=['scrapy_fake_useragent.providers.FakeUserAgentProvider']The package has alsoFakerProvider(powered byFaker library) andFixedUserAgentProviderimplemented and available for use if needed.Each provider is enabled individually, and used in the order they are defined.
In case a provider fails execute (for instance, it canhappento fake-useragent because of it’s dependency
with an online service), the next one will be used.Example of whatFAKEUSERAGENT_PROVIDERSsetting may look like in your case:FAKEUSERAGENT_PROVIDERS=['scrapy_fake_useragent.providers.FakeUserAgentProvider','scrapy_fake_useragent.providers.FakerProvider','scrapy_fake_useragent.providers.FixedUserAgentProvider','mypackage.providers.CustomProvider']Configuring fake-useragentParameter:FAKE_USERAGENT_RANDOM_UA_TYPEdefaulting torandom.Other options, as example:firefoxto mimic only firefox browsersdesktopormobilevalues to send desktop or mobile strings respectively.You can also set theFAKEUSERAGENT_FALLBACKoption, which is afake-useragentspecific fallback. For example:FAKEUSERAGENT_FALLBACK='Mozilla/5.0 (Android; Mobile; rv:40.0)'What it does is, if the selectedFAKE_USERAGENT_RANDOM_UA_TYPEfails to retrieve a UA, it will use
the type set inFAKEUSERAGENT_FALLBACK.Configuring fakerParameter:FAKER_RANDOM_UA_TYPEdefaulting touser_agentwhich is the way of selecting totally random User-Agents values.
Other options, as example:chromefirefoxConfiguring FixedUserAgentIt also comes with a fixed provider (only provides one user agent), reusing the Scrapy’s defaultUSER_AGENTsetting value.Usage withscrapy-proxiesTo use with middlewares of random proxy such asscrapy-proxies, you need:setRANDOM_UA_PER_PROXYto True to allow switch per proxyset priority ofRandomUserAgentMiddlewareto be greater thanscrapy-proxies, so that proxy is set before handle UALicenseThe package is under MIT license. Please seeLICENSE.