Package: proxyup
PROXYUPProxyUP is a package designed for retrieving proxies from a public API. Transparently, it will only retrieve valid proxies,
checking them constantly at a fixed rate and ensuring the proxies answer before their delivery.InstallationIt can be installed through pip:pipinstallproxyupBasic UsageA simple example of use to retrieve a single http proxy:fromproxyupimportProxyupRetrieverproxies=ProxyupRetriever()proxies.start()try:proxy=proxies.get_once()finally:proxies.close()print(proxy)'http://X.X.X.X:XXXX'As soon as the retriever is instantiated, it begins to scrap proxies in the background at a fixed rate. Internally it will hold a list of valid proxies that are periodically checked and updated.Theget_once()method allows to retrieve N proxies in a single-shot. By default, only one proxy is retrieved.It is encouraged to wrap theProxyupRetrieverin a context manager:fromproxyupimportProxyupRetrieverwithProxyupRetriever(proxy_type="http")asproxies:# Valid proxy types=["http", "socks4", "socks5"]proxies_list=proxies.get_once(4)print(proxies_list)['http://X.X.X.X:XXXX','http://X.X.X.X:XXXX','http://X.X.X.X:X','http://X.X.X.X:X']All the returned proxies have passed properly the control measures. These measures consists of the following rules:They all had a server listening on the specified port.They all answered with a 200 status code when requestedhttps://www.google.comthrough them.They all were responsive in the last60seconds. This parameter is modificable throguh thecheck_interval_secondsduring instantiation of the class.Advanced usageIf desired, it can be wrapped in an infinite iterator that retrieves X number of proxies as follows:fromproxyupimportProxyupRetrieverwithProxyupRetriever(proxy_type="http")asproxies:# Valid proxy types=["http", "socks4", "socks5"]forproxies_listinproxies[4]:# The index es the size of the list to retrieve in a single shotprint(proxies_list)['http://X.X.X.X:XXXX','http://X.X.X.X:XXXX','http://X.X.X.X:X','http://X.X.X.X:X']['http://X.X.X.X:XXXX','http://X.X.X.X:XXXX','http://X.X.X.X:X','http://X.X.X.X:X']['http://X.X.X.X:XXXX','http://X.X.X.X:XXXX','http://X.X.X.X:X','http://X.X.X.X:X']...This iterator will run forever, reporting valid proxies on each iteration, which may be the same or different proxies than the previous iteration.If a valid proxy is detected to not be valid anymore, it will never be yielded again.
The internal proxy list is constantly being updated at a rate of120seconds, a value that can be modified by using theupdate_interval_secondsparameter.A single update will scrap around 100-200 new proxies to include in the proxies list. Previous proxies are not removed unless they are detected to not be valid anymore.In order to avoid an internal list overflow, a limit is specified in the number of internal max proxies allowed to be kept for checks. This value is by default 1000 proxies, but it can be modified through the parameterproxy_cache_size.Note that it is important to close the proxies object. Otherwise, their internal threads will not know when to finish and will run in background forever, avoiding the process termination.ReferencesThis package, as of version 0.0.1, uses the API fromhttps://proxyscrape.com/to scrap new proxies. Note that this backend might change in future releases of the package.