Package: ua-spoofer
ua_spooferA Python module which collects, lists, and returns up to date and commonly used
User Agent strings. This can be helpful for avoiding fingerprinting, and
bypassing anti-bot/scraping measures. It also provides aRequestssession wrapper which automatically
uses a random user agent on every connection.User AgentsAuser agentstring is sent as a
header in HTTP requests to identify which browser and operating system the
client is using. It can be used by websites to tailor the content to the device
and software a visitor is using. It can also be used to block or restrict
certain programs' access, such as bots, web crawlers and scrapers. Another
consequence of these strings is they can help build a profile of a user, using
the unique compination of browser and operating system versions, a technique
calledfingerprinting.User agent spoofing replaces the user agent string with a random one from a
list of common strings, disguising the type of client from the server and
making it harder to track the user between requests. This is one of the ways to
bypass restrictions and mitigate against fingerprinting.DetailsA problem with similar modules and programs is they either use a static
dataset, or scrape user agents from sources which are either badly outdated or
completely broken. ua_spoofer attempts to solve this by fetching data which is
up to date, based on the latest browser versions, and also amalgamates data
from several sources. This provides redundancy and a good mix of current user
agents, without depending on an API or downloading a static dataset which
quickly goes out of date. More sources can be added over time without breaking
compatibility.Installingua_spoofer requires Python 3, plusRequestsandBeautifulSoup, commonly
used modules for scraping purposes.pip install ua_spooferUsingGetting User Agentsfrom ua_spoofer import UserAgent

ua = UserAgent()

# Random user agents from a specified browser    
ua.chrome
ua.firefox
ua.ie

# Any random user agent
ua.random

# Get a list of supported browsers
ua.BROWSERS

# Get the list of all user agent strings
ua.all

# Update the list
ua.update()Using the Requests Session wrapperfrom ua_spoofer import SpoofSession

s = SpoofSession()

# Each request will use a different user agent string
# A few other headers are randomised too
# To demonstrate:
s.get("https://icanhazheaders.com/").json()
s.get("https://icanhazheaders.com/").json()
s.get("https://icanhazheaders.com/").json()

# To get the UserAgent instance of the session
s.ua

# Updating the user agent list is done as you would expect
s.ua.update()Other projectsAs mentioned earlier, there are other Python modules which attempt to do
similar things:fake-useragentrequests-random-user-agentrandom_user_agentUser agent spoofing isn't the only technique to bypass restrictions, with more
sites being Javascript based and using more aggressive techniques to protect
against crawlers, bots and DDoS attacks, sometimes other methods are necessary,
including headless browser automation.cloudflare-scrapeis a module
to bypass Cloudflare's anti-bot systemPhantomJSis a scriptable headless browserSeleniumis a full browser automation frameworkScrapyis a Python framework for building crawlersSpynneris another scriptable Python
browser moduleIn some cases, Tor or a VPN can be used to hide the client's IP address for
proper anonymity.Licenseua_spoofer is released under the terms of the Apache 2.0 license.