Package: greendeck-proxygrabber
greendeck-proxygrabber 🎭This package is developed byGreendeckInstall from piphttps://pypi.org/project/greendeck-proxygrabber/pip install greendeck-proxygrabberWHATS NEW?Added proxy grabbing support of 4 new regions to proxy service, proxy grabber and proxy scraper.👉 What is proxy service?Proxy service is a service that keeps and updates a Mongo Database with latest up and running proxies.👉 How to use?import the service classfromgreendeck_proxygrabberimportProxyServiceservice=ProxyService(MONGO_URI='mongodb://127.0.0.1:27017',update_time=300,pool_limit=1000,update_count=200,database_name='proxy_pool',collection_name_http='http',collection_name_https='https',country_code='ALL')This creates a service object.Argsupdate_time = Time after which proxies will be updated (in seconds)pool_limit = Limit after which insertion will change to updatingupdate_count = Number of proxies to request grabber at a timedatabase_name = Mongo Database name to store proxies incollection_name_http = Collection name to store http proxies incollection_name_https = Collection name to store https proxies incountry_code = ISO code of one of regions supportedList of supported regions is:Combined Regions: ALLUnited States: USGermany: DEGreat Britain: GBFrance: FRCzech Republic: CZNetherlands: NLIndia: INStarting the serviceservice.start()Starting service gives the following output:MONGO_URI: mongodb://127.0.0.1:27017
Database: proxy_pool
Collection names: http, https
Press Ctrl+C once to stop...
Running Proxy Service...This will run forever and will push/update proxies in mongodb after every {update_time} seconds.👉 What is proxy to mongo?Proxy to mongo is a functionality that lets you grab a set of valid proxies from the Internet and store it to the desired MongoDB database. You can schedule this to update or insert a given set of proxies to your database of pool, i.e. put it on airflow or any task scheduler.👉 How to use?import the ProxyToMongo classfromgreendeck_proxygrabberimportProxyServiceservice=ProxyToMongo(MONGO_URI=MONGO_URI,pool_limit=1000,length_proxy=200,database_name='proxy_pool',collection_name_http='http',collection_name_https='https',country_code='DE')This creates a service object.Argspool_limit = Total number of proxies to keep in mongo/pass None if you don't want to updatelength_proxy = Number of proxies to fetch at oncedatabase_name = Mongo Database name to store proxies incollection_name_http = Collection name to store http proxies incollection_name_https = Collection name to store https proxies incountry_code = ISO code of one of regions supportedList of supported regions is:Combined Regions: ALLUnited States: USGermany: DEGreat Britain: GBFrance: FRCzech Republic: CZNetherlands: NLIndia: INCalling the ProxyToMongo grabberservice.get_quick_proxy()Starting Grabber gives the following output:MONGO_URI: mongodb://127.0.0.1:27017
Database: proxy_pool
Collection names: http, https
Press Ctrl+C once to stop...
Running Proxy Grabber...This will run forever and will push/update proxies in mongodb after every {update_time} seconds.👉 How to use Proxy Grabber Class?importProxyGrabberclassfromgreendeck_proxygrabberimportProxyGrabberinitializeProxyGrabberobjectgrabber=ProxyGrabber(len_proxy_list,country_code,timeout)Here default values of some arguments are,len_proxy_list = 10
country_code = 'ALL'
timeout = 2Currently the program only supports proxies of combined regionsGetting checked, running proxiesThe grab_proxygrab_proxy()function helps to fetch the proxies.grabber.grab_proxy()This returns a dictionary of the following structure:{'https':[<listofhttpsproxies>],'http':[<listofhttpproxies>],'region':'ALL'# default for now}Getting an unchecked list of proxiesThe grab_proxyproxy_scraper()method ofScrapeProxyhelps to fetch the proxies.
This returns a list of 200 proxies of both type http and https.fromgreendeck_proxygrabberimportScrapeProxyproxies_http,proxies_https=ScrapeProxy.proxy_scraper()This returns list of proxies of type http proxies followed by https proxies.http_proxies = [< list of http proxies >]
https_proxies = [< list of https proxies >]Filtering invalid proxies from a list of proxiesTheproxy_checker_httpsandproxy_checker_httpmethods fromProxyCheckerclass helps to validate the proxies.Given a list of proxies, it checks each of them to be valid or not, and returns a list of valid proxies from the proxies feeded to it.fromgreendeck_proxygrabberimportProxyCheckervalid_proxies_http=ProxyChecker.proxy_checker_http(proxy_list=proxy_list_http,timeout=2)valid_proxies_https=ProxyChecker.proxy_checker_https(proxy_list=proxy_list_https,timeout=2)👉 How to build your own pip packageopen an account herehttps://pypi.org/In the parent directorypython setup.py sdist bdist_wheeltwine upload dist/*referenceshttps://medium.com/small-things-about-python/lets-talk-about-python-packaging-6d84b81f1bb5