Package: proxyscrape
A library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5). Supports Python 2.7+ and 3.4+.NOTE: This library isn’t designed for production use. It’s advised to use your own proxies or purchase a service which
provides an API. These are merely free ones that are retrieved from sites and should only be used for development
or testing purposes.importproxyscrapecollector=proxyscrape.create_collector('default','http')# Create a collector for http resourcesproxy=collector.get_proxy({'country':'united states'})# Retrieve a united states proxyInstallationThe latest version of proxyscrape is available viapip:$pipinstallproxyscrapeAlternatively, you can download and install from source:$pythonsetup.pyinstallProvided ProxiesCurrent proxies provided are scraped from various sites which offer free HTTP, HTTPS, SOCKS4, and SOCKS5 proxies; and
don’t require headless browsers or selenium to retrieve. The list of sites proxies retrieved are shown below.resourceresource typeurlanonymous-proxyhttp, httpshttps://free-proxy-list.net/anonymous-proxy.htmlfree-proxy-listhttp, httpshttps://free-proxy-list.netproxy-daily-http
proxy-daily-socks4
proxy-daily-socks5http
socks4
socks5http://www.proxy-daily.comsocks-proxysocks4, socks5https://www.socks-proxy.netssl-proxyhttpshttps://www.sslproxies.orguk-proxyhttp, httpshttps://free-proxy-list.net/uk-proxy.htmlus-proxyhttp, httpshttps://www.us-proxy.orgSeeIntegrationsection for additional proxies.Getting StartedProxy Scrape is a library aimed at providing an efficient an easy means of retrieving proxies for web-scraping
purposes. The proxies retrieved are available from sites providing free proxies. The proxies provided, as shown in the
above table, can be of one of the following types (referred to as aresource type): http, https, socks4, and socks5.CollectorsCollectors serve as the interface to retrieving proxies. They are instantiating at module-level and can be retrieved
and re-used in different parts of the application (similar to the Pythonlogginglibrary). Collectors can be created
and retrieved via thecreate_collector(…)andget_collector(…)functions.fromproxyscrapeimportcreate_collector,get_collectorcollector=create_collector('my-collector',['socks4','socks5'])# Some other section of codecollector=get_collector('my-collector')Each collector should have a unique name and be initialized only once. Typically, only a single collector of a given
resource type should be utilized. Filters can then be applied to the proxies if specific criteria is desired.When given one or more resources, the collector will use those to retrieve proxies. If one or more resource types
are given, the resources for each of the types will be used to retrieve proxies.Once created, proxies can be retrieved via theget_proxy(…)or theget_proxies(…)function. This optionally takes afilter_optsparameter which can filter by the following:code(us, ca, …)country(united states, canada, …)anonymous(True, False)type(http, https, socks4, socks5, …)fromproxyscrapeimportcreate_collectorcollector=create_collector('my-collector','http')# Retrieve any http proxyproxy=collector.get_proxy()# Retrieve only 'us' proxiesproxy=collector.get_proxy({'code':'us'})# Retrieve only anonymous 'uk' or 'us' proxiesproxy=collector.get_proxy({'code':('us','uk'),'anonymous':True})# Retrieve all 'ca' proxiesproxies=collector.get_proxies({'code':'ca'})Filters can be applied to every proxy retrieval from the collector viaapply_filter(…). This is useful when the same
filter is expected for any proxy retrieved.fromproxyscrapeimportcreate_collectorcollector=create_collector('my-collector','http')# Only retrieve 'uk' and 'us' proxiescollector.apply_filter({'code':'us'})# Filtered proxiesproxy=collector.get_proxy()# Clear filtercollector.clear_filter()Note that some filters may instead use specific resources to achieve the same results (i.e. ‘us-proxy’ or ‘uk-proxy’ for
‘us’ and ‘uk’ proxies).Blacklists can be applied to a collector to prevent specific proxies from being retrieved. They accept either one or more Proxy
objects, or a host + port number combination and won’t allow retrieval of matching proxies. Proxies can be individually removed
from blacklists or the entire blacklist can be cleared.fromproxyscrapeimportcreate_collectorcollector=create_collector('my-collector','http')# Add proxy to blacklistcollector.blacklist_proxy(Proxy('192.168.1.1','80',None,None,None,'http','my-resource'))collector.blacklist_proxy(host='192.168.1.2',port='8080')# Blacklisted proxies won't be includedproxy=get_proxy()# Remove individual proxiescollector.remove_blacklist(host='192.168.1.1',port='80')# Clear blacklistcollector.clear_blacklist()Instead of permanently blacklisting a particular proxies, a proxy can instead be removed from internal memory. This
allows it to be re-added to the pool upon a subsequent refresh.fromproxyscrapeimportcreate_collectorcollector=create_collector('my-collector','http')# Remove proxy from internal poolcollector.remove_proxy(Proxy('192.168.1.1','80',None,None,'http','my-resource'))Apart from automatic refreshes when retrieving proxies, they can also be forcefully refreshed via therefresh_proxies(…)function.fromproxyscrapeimportcreate_collectorcollector=create_collector('my-collector','http')# Forcefully refreshcollector.refresh_proxies(force=True)# Refresh only if proxies not refreshed within `refresh_interval`collector.refresh_proxies(force=False)ResourcesResources refer to a specific function that retrieves a set of proxies; the currently implemented proxies are all
retrieves from scraping a particular web site.Additional user-defined resources can be added to the pool of proxy retrieval functions via theadd_resource(…)function. Resources can belong to multiple resource types.fromproxyscrapeimportadd_resourcedeffunc():return{Proxy('192.168.1.1','80','us','united states',False,'http','my-resource'),}add_resource('my-resource',func,'http')As shown above, a resource doesn’t necessarily have to scrape proxies from a web site. It can be return a hard-coded
list of proxies, make a call to an api, read from a file, etc.The set of library- and user-defined resources can be retrieved via theget_resources(…)function.fromproxyscrapeimportget_resourcesresources=get_resources()Resource TypesResource types are groupings of resources that can be specified when defining a collector (opposed to giving a
collection of resources.Additional user-defined resource types can be added via theadd_resource_type(…)function. Resources can optionally
be added to a resource type when defining it.fromproxyscrapeimportadd_resource_typeadd_resource_type('my-resource-type')add_resource_type('my-other-resource-type','my-resource')# Define resources for resource typeThe set of library- and user-defined resource types can be retrieved via theget_resource_types(…)function.fromproxyscrapeimportget_resource_typesresources=get_resource_types()IntegrationIntegrations are proxy implementations that are specific to a particular website or API and have a distinctively
separate use case.ProxyScrapeTheProxyScrape.com APIprovides a means of accessing thousands of proxies of various types (HTTP, SOCKS4, SOCKS5) in
an efficient manner. These are vetted and validated with a minimal response time.Theget_proxyscrape_resource(…)function is used to dynamically create a new resource for using the proxyscrape API.
The resource name can then be added to a resource type and used like any other library- or user-defined resource. The
following parameters are used for the API:proxytype(http, socks4, socks5, all)timeout(milliseconds > 0)ssl(yes, no, all)anonymity(elite, anonymous, transparent, all)country(any Alpha 2 ISO country code, all)fromproxyscrapeimportget_proxyscrape_resourceresource_name=get_proxyscrape_resource(proxytype='http',timeout=5000,ssl='yes',anonymity='all',country='us')ContributionContributions or suggestions are welcome! Feel free toopen an issueif a bug is found or an enhancement is desired,
or even apull request.ChangelogAll changes and versioning information can be found in theCHANGELOG.LicenseCopyright (c) 2018 Jared Gillespie. SeeLICENSEfor details.