Package: aproxyrelay
#       ######                                ####### #      #     # #####   ####  #    # #   #    #     # ###### #        ##   #   ##   #     #     # #    # #    #  #  #   # #     #     # #      #       #  #   # ##     #    ######  #    # #    #   ##     #      ######  #####  #      #    #   ########    #       #####  #    #   ##     #      #   #   #      #      ######   ##     #    #       #   #  #    #  #  #    #      #    #  #      #      #    #   ##     #    #       #    #  ####  #    #   #      #     # ###### ###### #    #   #Byundeƒined-------------------------------------AProxyRelay: An Async Request Library with Proxy RotationAProxyRelay is an asynchronous request library designed for easy data retrieval using various proxy servers. It seamlessly handles proxy rotation, preserves data that fails to be requested, and simplifies API scraping. The library is written inPython 3.12.2.In addition, tested proxies will be shared with other people using this library. The more this library is utilized, the bigger the pool of available proxies.Our scraper, used to obtain proxies, is highly modular and plug-and-play, making it easy to contribute to.UsageAProxyRelay streamlines the process of making asynchronous requests with proxy servers. It offers the following features:Asynchronously fetches lists of free proxies from various sources based on the provided zoneTests and shares proxies with other users of the libraryIdentifies and discards bad proxies, preserving data for failed target requestsBypasses API limiters in an asynchronous manner (for educational purpose)Example# -*- mode: python ; coding: utf-8 -*-fromaproxyrelayimportAProxyRelay# Note: Duplicates will be removed by the librarytargets=['https://gg.my-dev.app/api/v1/proxies/available?zone=US&anonimity=all&protocol=all&page=1&size=100&type=example','https://gg.my-dev.app/api/v1/proxies/available?zone=DE&anonimity=all&protocol=all&page=1&size=100&type=example','https://gg.my-dev.app/api/v1/proxies/available?zone=NL&anonimity=all&protocol=all&page=1&size=100&type=example','https://gg.my-dev.app/api/v1/proxies/available?zone=CA&anonimity=all&protocol=all&page=1&size=100&type=example','https://gg.my-dev.app/api/v1/proxies/available?zone=AU&anonimity=all&protocol=all&page=1&size=100&type=example',]# Initialize proxy relayproxy_relay=AProxyRelay(targets=targets,timeout=5,scrape=True,filter=True,zones=['us'],unpack=lambdadata,target:data['results'],debug=False,)# Fetch datadata=proxy_relay.start()# Result Queueprint(data.qsize())whilenotdata.empty():content=data.get()print(content)A Proxy Relay: InstallationSimply runpip install aproxyrelayParametersParametersTypeFunctionDescriptiontargetslist[str]Target endpoints provided in an arrayEach endpoint will be requested with an available proxy. If a proxy is unavailable and the request fails, we store it in a queue and try it out with another proxy until we have data.timeoutintAllowed proxy timeout.Defaults to 5A proxy has to respond within the provided timeout to be considered valid. Otherwise, it will be discarded.scrapeboolIndicator to utilize the proxy scraper.Defaults to TrueThe decision to scrape for proxies is determined by the value of this parameter. When set to True (default), the proxy scraper is used, which is slower but provides a broader range of proxies. When set to False, proxies are fetched from a single source, offering a faster but more limited selection.filterboolIndicator for filtering bad proxies.Defaults to TrueIf set to True (default), the tool will test proxy connections before using them. This process might take a bit longer, but it ensures that the proxies are valid before utilization.zoneslist[str]An array of proxy zones.Defaults to ['US']Sometimes it matters where the proxy is located. Each item in this list ensures the proxy is located in that specific zone, and requests made from the proxy are coming from the location provided. It acts like a whitelist for allowed proxy locations.unpacklambdaAnonymous function for unpacking data.Defaults tolambda data, target: dataWhen a request has been made to a target through a proxy and data has been fetched, this lambda method formats the result data before putting it into the result queue.data-> output from the target,target-> target URL.debugboolIndicator which enables debug mode.Defaults to FalseWhen true, additional logging will be printed to the terminal, enabling debug mode.A Proxy Relay: Local DevelopmentTo install all library dependencies for local development, excluding the core code available locally, use the following command within a virtual environment:pip install -e .[dev]This command installs dependencies and removes the core code of AProxyRelay from the virtual environment.Contribute to AProxyRelayAProxyRelay encourages contributions to enhance its capabilities by allowing users to create custom scrapers. These scrapers are designed to fetch proxy data from different sources. The process is straightforward, and here's a guide to help you get started:Scraper StructureAProxyRelay includes a dedicated folder namedscrapers. Within this folder, each scraper, prefixed withparser_, inherits from theparser.pyfile. To provide you with a general understanding, here's an example of theMainScraperclass:fromqueueimportQueuefrom.coreimportScraperCoreclassMainScraper(ScraperCore):def__init__(self)->None:ScraperCore.__init__(self)self.zone=None@classmethodasyncdefformat_url(cls,url,*args,**kwargs)->str:"""Formats URL before scraping, let us adjust query parameters for each parser"""cls.zone=kwargs.get("zone","us")new_url=f'{url}'returnnew_url@classmethodasyncdefformat_raw(cls,html:str):"""Parse text/html pages, customized method for the parser of this website"""raiseNotImplementedError('Format raw parser has not been implemented yet')@classmethodasyncdefformat_data(cls,data:dict,queue:Queue):"""Data formatter, formats data and returns is back in the process Queue"""raiseNotImplementedError('Form data parser has not been implemented yet')Creating your own Proxy ScraperTo contribute your own proxy scraper, follow these steps:Create a new parser class inside the scrapers folderInherit from theMainScraper.Overwrite the necessary methods required for scraping additional proxy servers.Methods to Overwrite:format_url: Manipulate the proxy list request URL before making a request, enabling adjustment of various query parameters.format_raw: When the data obtained from the link istxt/html, this method should scrape the data and format it into workable data.format_data: This method is triggered when the call to the proxy list returns a dictionary, or when format_raw has been completed.Formatting Data:Your goal is to format the data in theformat_datamethod and place it into the provided Queue. The data should be structured as follows:data={"zone":"US","method":"http","anonymity":"anonymous","protocol":"http","port":"8080","ip":"127.0.0.1",}queue.put(data)CongratulationsIf done correctly, congratulations! You've successfully created a new proxy parser for this library.Add the targeted link and your scraper toscrapers/__init__.py.Feel free to contribute, share your improvements, and expand the library's capabilities. Your efforts contribute to a growing pool of available proxies for the AProxyRelay community.Compiling to packageTo compile the library into a package, use the following command after installingrequirements.txt:pythonsetup.pysdistbdist_wheelThis will generate the package in thedistfolder.Note: A custom version can be set with the environment variableCUSTOM_VERSIONVersioningThe version public onPyPicontains a version number based on the pipeline builds and looks like:` aproxyrelay 1.104.1rc7696232336 `Breaking the version down:aproxyrelay: The name of the package1: Major version104: Build number1: Build retry numberrc7696232336: Random build number