Saravanan666/pypi-org huggingface dataset:
Package: proxyenv
This is a python module that starts aSquidproxy running in a docker
container, and sets up an environment for using this proxy to handle
HTTP(S) requests both from python code and external programs. The
purpose is mainly for testing proxy support in code that needs to make
HTTP(S) requests. The proxy supports both HTTP and HTTPS, and Basic
proxy authorization.Using an existing HTTP proxyIf a proxy is already set in thehttp_proxyenvironment variable,
the newly created squid proxy will still work by using the existing
proxy as “cache peer”. If you don’t want this, deletehttp_proxyfrom the environment before starting the temporary proxy.Usagefrom proxyenv.proxyenv import ProxyFactory
proxyfact=ProxyFactory()

# Example for using urllib
from six.moves.urllib import request
with proxyfact() as proxy:
    opener  = request.build_opener(proxy.get_handler)
    opener.open(url)

# Example for starting external program with proxy-related environment
# variables
from subprocess import check_output
with proxyfact(user="johndoe:secret", port=3077) as proxy:
    check_output(["/usr/bin/wget", url])Please use themainprogram inproxyenv/proxyenv.pyas
additional source of documentation.The proxyenv scriptThe package also contains a console scriptproxyenv.usage: proxyenv [-h] [-p PORT] [-u USER] [-i IMPL] [-s] [-w] [-v]
                [URL [URL ...]]

positional arguments:
  URL                   an URL to retrieve

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  proxyport to use (default: 3128)
  -u USER, --user USER  proxy user in user:password format
  -i IMPL, --impl IMPL  set implementaton (cmdline or api, default: api)
  -s, --shell           open a shell in new environment
  -w, --wait            wait for input
  -v, --verbose         verbose output
Package: proxy-factory
自动从网上抓取免费代理,并对代理的可用性和匿名性进行检查，同时定时检查有效代理和无效代理，对于多次检查始终无效的代理，做放弃处理。同时检查函数可以自定义指定，用来针对不同的检查结果做出不同的反应。当然代理网站也可以自定义，简单的几行代码几条配置信息，实现最大限度的free-style。INSTALL# python3 以上版本
pip install proxy-factory
# 依赖 redis(必须), tesseract-ocr（可选）USAGEmashichaodeMac-mini:toolkit mashichao$ product -h
usage: product [-h] [-cm CHECK_METHOD] [-sm SPIDER_MODULE] [--console]
               [--console-host CONSOLE_HOST] [--console-port CONSOLE_PORT]
               [-s SETTINGS] [-ls LOCALSETTINGS] [-d]
               [{stop,start,restart,status}]

positional arguments:
  {stop,start,restart,status}

optional arguments:
  -h, --help            show this help message and exit
  -cm CHECK_METHOD, --check-method CHECK_METHOD
                        proivde a check method to check proxies.
                        eg:module.func
  -sm SPIDER_MODULE, --spider-module SPIDER_MODULE
                        proivde a module contains proxy site spider methods.
                        eg:module1.module2
  --console             start a console.
  --console-host CONSOLE_HOST
                        console host.
  --console-port CONSOLE_PORT
                        console port.
  -s SETTINGS, --settings SETTINGS
                        Setting module.
  -ls LOCALSETTINGS, --localsettings LOCALSETTINGS
                        Local setting module.
  -d, --daemon
####################################################################
- product start： 程序开始(阻塞式)
- product -d start: 程序开始(守护进程模式)
- product restart 程序重启(守护进程模式)
- product stop 程序关闭(守护进程模式)
- product status 程序状态(守护进程模式)
- product --console 开启一个console客户端，调试专用，详细请参见(https://github.com/ShichaoMa/toolkit)
- product -s settings 指定一个配置模块。(只要在sys.path中就可以找到)
- product -ls localsettings 指定一个自定义配置模块。(只要在sys.path中就可以找到)
- product -cm check-method 指定一个自定义检查方法。(只要在sys.path中就可以找到)
- product -sm spider-module 指定一个自定义的spider模块，存放自定义的spider方法。(只要在sys.path中就可以找到)DOCKERdockerrun-d-eREDIS_HOST=192.168.200.150cnaafhvk/proxy-factoryproductstartCONFIGCUSTOM CHECKdefcheck(self,proxy):"""自义定检查方法:param self: ProxyFactory对象:param proxy: 代理:return: True则代理可用，否则False"""importrequestsresp=requests.get("http://2017.ip138.com/ic.asp",proxies={"http":"http://%s"%proxy})self.logger.info(resp.text)....returnresp.status_code<300CUSTOM PROXY SITE METHODdeffetch_custom(self,page=5):"""自定义代理网站抓取:param self:ProxyFactory对象:param page: 可以在里记录一些可选参数，但是方法只能接收一个必选参数:return: set类型的代理列表，ip:port"""proxies=set()url_tmpl="http://www.kxdaili.com/dailiip/1/%d.html"forpage_numinrange(page):url=url_tmpl%(page_num+1)soup=BeautifulSoup(get_html(url,self.headers),"html")table_tag=soup.find("table",attrs={"class":"segment"})trs=table_tag.tbody.find_all("tr")fortrintrs:tds=tr.find_all("td")ip=tds[0].textport=tds[1].textlatency=tds[4].text.split(" ")[0]iffloat(latency)<0.5:# 输出延迟小于0.5秒的代理proxy="%s:%s"%(ip,port)proxies.add(proxy)returnproxiesSETTINGSREDIS_HOST="0.0.0.0"REDIS_PORT=6379# 质量不好的代理检查的时间间隔BAD_CHECK_INTERVAL=60# 质量不好的代理连续检查失败次数的最大值，超过则丢弃FAILED_TIMES=5# 质量好的代理检查的时间间隔GOOD_CHECK_INTERVAL=60# 抓取新代理的时间间隔FETCH_INTERVAL=60参考资料一键获取免费真实的匿名代理
Package: proxygen-cli
proxygen-cliInstallationShould be as simple aspip install proxygen-cliThe python package includes an execuatableproxygen.
Typeproxygento see a list of available commands.ConfigurationCredentialsThe CLI has its own client credentials, which need to be input.
Contactdeathstarsquad or theplatforms-api-producer-supportslack channel to find out what they are.All users should also have individual credentials.proxygen-clineeds to know about them.Simply execute the following command which will prompt you to enter yourclient_id,client_secret,username, andpassword:proxygen credentials setIf you need to update any credentials in the future, use the following command:proxygen credentials set <KEY> <VALUE>Settingsproxygen-clineeds to know what API you are developing.proxygen settings set api <API-NAME>Your user must have permissions to manipulate instances/secrets/specs for the API you set here.
If you do not have sufficient permissions commands will fail.
If you believe your permissions are incorrect, contact theplatforms-api-producer-supportchannel.CommandsCommands are documented inside the CLI itself.
Typeproxygento see a list of available commands.
Package: proxygetter
proxygetterDescriptionproxygetteris a Python library that provides a fast and customizable way to scrape, filter, and manage proxies. It's powered by asyncio and aiohttp to validate proxies asynchronously. Originally designed to scrape fromsslproxies.org, it now supports customizable sources and multiple filters.InstallationInstall the package via pip:pipinstallproxygetterFeaturesProxy ManagementManage your proxies with ease using theProxyManagerclass.fromproxygetterimportProxyManagermanager=ProxyManager()Asynchronous FilteringFilter proxies based on their validity asynchronously.valid_proxies=manager.filter_with_validity(url="http://example.com")Proxy InformationAccess details about each proxy through theProxyclass.proxy=valid_proxies[0]print(proxy.get_requests_format())print(proxy.get_selenium_format())Advanced FiltersGet proxies using advanced filters like country code, anonymity, https support, Google compatibility, and last checked time.filtered_proxies=manager.get_proxies(country_code='US',anonymity='elite proxy',https=True,google=True,last_checked_max=600)Fetch a Random ProxyYou can fetch a random proxy based on specified filters.random_proxy=manager.get_random_proxy(country_code='US',https=True)Environment ConfigurationConfigure default user agent and timeout using environment variables.exportPROXY_URL_CHECKER_USER_AGENT=your_user_agentexportPROXY_URL_CHECKER_TIMEOUT=your_timeout_valueLicenseThis project is under the MIT License - see theLICENSEfile for details.Upcoming FeaturesProxy blacklistingAdditional proxy databasesEnhanced documentation and examplesFeel free to contribute or suggest improvements.
Package: proxygrab
ProxyGrabSoftware to scrap proxies for my web scrapping and other testing purposes.I made this software to scrap proxies for my web scrapping and other testing purposes. This program just usesaiohttpto get the response from API and return the proxies, also it can scrape proxies from a few sites so that it can be used without using the API.NOTE:This library isn't designed for production use. It's advised to use your own proxies or purchase a service which provides an API. These are merely free ones that are retrieved from sites and should only be used for development or testing purposes.InstallationThe latest version of proxygrab is available viapip:pipinstall--upgradeproxygrabProvided ProxiesProviderProxy Types avaiableUrlProxyscrapehttp, https, socks4, socks5https://proxyscrape.com/Proxy-Listhttp, https, socks4, socks5https://www.proxy-list.download/SSL Proxieshttpshttps://www.sslproxies.org/Free Proxy Listhttp, httpshttps://free-proxy-list.net/US Proxieshttp, httpshttps://www.us-proxy.org/Socks Proxysocks4, socks5https://www.socks-proxy.net/Documentationhttps://proxygrab.divkix.meContribuitingWanna help and improve this project?Make sure to follow these before opening a PR:Make sure your PR passes the test and is formatted according to pre-commit.Make sure the package is working without any issues!LicenseThis project is licensed under the MIT License - see theLICENSEfile for detailsThanks to@JaredLGillespiefor hisproxyscrape libraryfrom which I took scrappers!Proxy Providers mentioned above
Package: proxy-grabber
Proxy-GrabberSimple http proxy grabber and checkerInstallation$ pip3 install proxy-grabberUsagefromproxy_grabberimportProxyGrabbergrabber=ProxyGrabber()# --- Grabbing proxies ---# Parse proxy from different sources# You can call generate_proxy_list() without arguments if you want to grab as more proxies as possiblegrabber.grab_proxies(proxy_limit=100)# --- Adding proxies ---# Notice: you can add proxies without grabbing# You can add some proxies from the filegrabber.load('./data/proxy.list')# Or you can add proxy manuallygrabber.add_proxies(['ip:port','ip:port',...])# --- Checking proxies ---# [optional]# You can specify proxy countriesgrabber.set_countries(['US','RU','CA',...])grabber.check_proxies()# --- Get results ---grabber.get_proxy()# Random checked proxygrabber.get_checked_proxies()# All checked proxiesgrabber.save('./data/checked_proxies.list')# Save checked proxies to the file
Package: proxygrabkit
ProxyGrabKitA useful python package to obtain free proxies using several services.DescriptionYour Python package for retrieving internet proxy servers using free services.FeaturesGimmeProxy.com Integration: Fetch proxies from GimmeProxy service.ProxyRotator.com Integration: Fetch proxies using the ProxyRotator API.InstallationYou can install the package using pip:pipinstallproxygrabkitUsageFetching proxies from GimmeProxy.comfromproxygrabkitimportGimmeProxyClient# Declare a object of type GimmeProxyClientproxy_fetcher=GimmeProxyClient()# Get a random proxyproxy=proxy_fetcher.get_proxy()print(proxy.proxy)print(proxy.lastChecked)Fetching proxies from ProxyRotator.comfromproxygrabkitimportRotatingProxyClientproxy_fetcher=RotatingProxyClient(api_key='xxxxxxxxxxxxxxxxxx')proxy=proxy_fetcher.get_proxy()print(proxy.proxy)print(proxy.lastChecked)More examplesSee examples folder for more complex examplesFuture PlansSupport for <www.proxy-list.download>: Add functionality to retrieve proxies from this service.Support forhttps://www.sslproxies.org/: Incorporate support for SSL proxies from this source.Continuous Improvement and New Service Additions: Regularly update and add support for new proxy services.ContributionIf you encounter any issues or have enhancements to propose, feel free to open an issue or submit a pull request.LicenseThis package is licensed under the MIT License. See the LICENSE file for details.
Package: proxy-information
Proxy Information Checkerproxy_informationis a Python package for verifying and collecting information about proxy servers. With this package, you can easily check the status, response time, anonymity level, and geolocation of a given proxy server.FeaturesCheck the status of a proxy server.Measure the response time of a proxy server.Determine the anonymity level (Transparent, Elite, or Anonymous) of a proxy server.Retrieve the country and country code of a proxy server's location.InstallationYou can installproxy_informationusing pip:pipinstallproxy_informationUsagefromproxy_informationimportProxyInformation# Initialize a ProxyInformation instancechecker=ProxyInformation()# Verify a proxy server and get its informationproxy_info=checker.check_proxy("103.88.35.200:1080")# Print the resultsprint(proxy_info)ContributingWe welcome contributions from the community to help us improve and grow this project. If you would like to contribute, please follow these guidelines:Check if there are any open issues you would like to work on, or if you have a new idea or feature to suggest, create a new issue on theGitHub repository.Fork the repository and create a new branch for your contributions.Make your changes, ensuring that your code follows the project's coding style and conventions.Write clear, concise, and well-documented code and commit messages.Test your changes to ensure they work as expected.Submit a pull request to the main repository, explaining the purpose of your changes and referencing the related issue.We appreciate your contributions and look forward to working together to make this project even better.LicenseThis project is licensed under the MIT License. By contributing to this project, you agree to adhere to the terms and conditions outlined in theLICENSEfile. The MIT License is a permissive open-source license that allows you to use, modify, distribute, and even sell the software, provided you give appropriate credit and include the original license when redistributing the code. For more details, please refer to theLICENSEfile.AuthorThis project is authored by a dedicated developer who strives to create innovative software solutions. To explore more projects and get in touch, visit the following:Your feedback and collaboration are greatly appreciated. Thank you for your interest in this project.ChangelogVersion 0.2:IP Retrieval Method:Switched toIPFyfor more reliable and unlimited public IP retrieval.Geolocation Service:Updated to useReallyFreeGeoIPfor improved accuracy and no request limits.Code Optimization:Reorganized and optimized code structure for better readability and maintainability.Error Handling:Implemented robust error handling to gracefully handle exceptions during API requests.Documentation:Improved function and class docstrings for better understanding.
Package: proxy-inspector
proxy-inspectorA Simple Tool for Validate ProxyUsage>>fromproxy_inspectorimportProxyInspector>>checker=ProxyInspector()>>proxies=checker.validate("https://www.google.com",["https://MY_PROXY_1","https://MY_PROXY_2","https://MY-PROXY-3"])>>print(proxies)["https://MY_PROXY_1","https://MY_PROXY_2","https://MY-PROXY-3"]
Package: proxylet
proxylet:  lightweight HTTP reverse proxy built on eventletThis module implements a lightweight reverse proxy for HTTP, using non-blockingIO based on the eventlet module.  It aims to do as little as possible whilesupporting simple request/response rewriting and being compatible with HTTPkeep-alive.Basic operation is via the 'serve' function, which will bind to thespecified host and port and start accepting incoming HTTP requests:proxylet.serve(host,port,mapper)Here 'mapper' is a function taking a proxylet.streams.HTTPRequest object,and returning either None (for '404 Not Found') or a 3-tuple giving thedestination host, destination port, and a rewriter object.The rewriter can be any callable that takes request and response streamsas arguments and returns wrapped versions of them, but it will most likelybe a subclass of proxylet.relocate.Relocator.  This class has the necessarylogic to rewrite the request for proxying.As an example of the available functionality, this mapping function willproxy requests to /svn to a private subversion server, requests to /filesto a private fileserver, and return 404 for any other paths:def mapper(req):svn = SVNRelocator("http://www.example.com/svn","http://svn.example.com/")if svn.matchesLocal(req.reqURI):return svn.mapping  # contains the (host,port,rewriter) tupleif req.reqURI.startswith("/files/"):return ("files.example.com",80,None)return None
Package: proxylist
Proxylist PackageThe proxylist package provides function and classes for:loading list of proxy servers from different sources like local file or network locationrotating proxy servers or picking them randomlyFeel free to give feedback in Telegram groups:@grablaband@grablab_ru.InstallationRun:pip install -U proxylistDocumentationDocumentation is available athttps://proxylist.readthedocs.ioUsage Example>>> from proxylist import ProxyList
>>> pl = ProxyList.from_local_file('var/proxy.txt')
>>> pl.get_random_server()
<proxylist.server.ProxyServer object at 0x7f1882d599e8>
>>> pl.get_random_server().address()
'1.1.1.1:8085'
>>> len(pl)
1000
Package: proxylists
proxylistsProxylists is a simple tool for getting a list of proxy IPs from
free proxy servers using several providers.Supported Providers:free-proxy-list.nethidemy.nameproxydb.netcan retrieve a list of IPs by provider or from all of them
using the "get_proxies()" method.DownloadThe list can be tabulated (one proxy by line, to save on .txt, csv, etc)
or a python dictionary.Usagefromproxylistimportget_proxiesproxies=get_proxies()Or get the list from a single provider:fromproxylist.proxiesimportProxyDB# getting from proxydb.netimportasyncioasyncdefget_proxies():returnawaitProxyDB().get_list()loop=asyncio.get_event_loop()proxies=loop.run_until_complete(get_proxies())print(proxies)LicenseProxylists is licensed under BSD license.
Package: proxylog
A simple HTTP proxy server that logs all client/server communications,
with some added features to parse the content such as automatic
gunzipping, output colorization and known data “pretty” formatting,
including XML, JSON, and YAML.ProjectHomepage:https://github.com/metagriffin/proxylogBugs:https://github.com/metagriffin/proxylog/issuesTL;DRInstall:$pipinstallproxylogProxyhttp://www.example.com/locally tohttp://localhost:8080/, log
all transactions to a file and display them colorized and formatted
on the console:$proxylog-rwww.example.com:80-p8080-otransactions.log-dcfDisplay a previous log file colorized and formatted:$proxylog-itransactions.log-dcfColorizingThe colorizing is done via thePygmentspackage; proxylog’s--themeoption is passed through to pygments, so any of the color
themes that pygments supports can be used. Use the following command
to list the available themes:$pygmentize-LstylesFormatting“Formatting” refers to proxylog’s ability to re-format known data
formats to a more human-friendly display, which when combined with
colorization, can yield very readable data.HOWEVER, this does
alter the actual data, so if you are using the data in any way,
you should not use the “–format” flag.Currently, the following formats are supported, with examples of how
they are “prettified”:XML:Input:<root  ><node   attr= "value">data</node>
  </root>Formatted output:<root>
  <node attr="value">data</node>
</root>JSON:Input:{"key": "value", "list": [3, "bar", null, "foo"]}Formatted output:{
  "key": "value",
  "list": [
    3,
    "bar",
    null,
    "foo"
  ]
}YAML:Input:{key: value, list: [3, bar, null, foo]}Formatted output:{
  "key": "value",
  "list": [
    3,
    "bar",
    null,
    "foo"
  ]
}
Package: proxy-machine
Proxy MachineDescriptionThe maximum number of proxies you can get is 35.000List of sites for parsing proxies:free-proxy-list.net/anonymous-proxy.htmlfree-proxy-list.netproxy-daily.comsslproxies.orgfree-proxy-list.net/uk-proxy.htmlus-proxy.orgapi.proxyscrape.comcheckerproxy.netproxy50-50.blogspot.comhidester.comawmproxy.netapi.openproxy.spacealiveproxy.comcommunity.aliveproxy.comhidemy.nameproxy11.comspys.meproxysearcherfatezeropubproxyproxylistsab57rushifty-httpsshifty-httpsunny9577multiproxyrootjazzproxyscan.ioproxy-list.downloadproxylistplus.comproxyhub.meproxylist4all.comproxynova.comxiladaili.comInstall through pippipinstallproxy_machineInstallsudoaptupdate&&sudoaptupgrade
sudoapt-getinstallpython3python3-pip
gitclonehttps://github.com/batiscuff/proxy_machinecdproxy_machine
pip3install-rrequirements.txtUsagepython3-mproxy_machineorproxy_machine--helpUsage with proxy checkerpython3-mproxy_machine-pcIn this case, all parsed proxies will pass through the
checker(this will take2-4 hours, prepare to wait) and
only working proxies will be written toproxies.txt.
However, remember that the main weakness of free proxies
is that they rapidly expire.Usage with other optionspython3-mproxy_machine-hFuture Development[x] - Add async checking of the proxy to improve timing.[x] - Improve cli options and args.[ ] - Upload to pypi.[ ] - Add proxy response time to the results by calculating execution in the checker[ ] - Add proxy location to the results[ ] - Add filtering and sorting options to the results[ ] - Add early stop, if the required number of proxies are reached with given constrains[ ] - Add more websites to retrieve proxiesLicenseThis project is GNUGeneral Public License v3.0licensed
Package: proxy-manager
This module loads and returns proxies for easy use withrequests.InstallationThis module is available via pip:$ pip install proxy-managerBasic Usageproxies.txt:00.11.222.33:4444
55.66.777.88:9999:username:passwordtest.py:fromproxymanagerimportProxyManagerproxy_manager=ProxyManager('proxies.txt')random_proxy=proxy_manager.random_proxy()print(random_proxy)# { 'http':'http://...', 'https':'https://...' }first_proxy=proxy_manager.next_proxy()print(first_proxy.get_dict())# { 'http':'http://00.11.222.33:4444', 'https':'https://00.11.222.33:4444' }second_proxy=proxy_manager.next_proxy()print(second_proxy.get_dict())# { 'http':'http://username:password@55.66.777.88:9999', 'https':'https://username:password@55.66.777.88:9999' }third_proxy=proxy_manager.next_proxy()print(third_proxy.get_dict())# { 'http':'http://00.11.222.33:4444', 'https':'https://00.11.222.33:4444' }DocumentationProxy File FormatProxies in proxy files must have one of the following formats:ip:portorip:port:username:passwordThese can be combined and alternated.ProxyManagerProxyManager(proxy_file_path)Returns a newProxyManagerinstance given the path to a proxy file.Parametersstring- File path to the proxyReturnsProxyManager- ProxyManager with proxies loadedExamplefromproxymngrimportProxyManagerproxy_manager=ProxyManager('proxies.txt')ProxyManager.random_proxyProxyManager.random_proxy()Returns a random proxy of those loaded into the proxy managerParametersnoneReturnsProxy- Random proxy as a Proxy objectExamplerandom_proxy=proxy_manager.random_proxy()print(random_proxy.get_dict())# { 'http':'http://...', 'https':'https://...' }ProxyManager.next_proxyProxyManager.next_proxy()Returns proxies consecutively. Thread-safe. Loops continuously through
available proxies, with wrapping.ParametersnoneReturnsProxy- Next proxy as a Project objectExamplefirst_proxy=proxy_manager.next_proxy()print(first_proxy.get_dict())# { 'http':'http://00.11.222.33:4444', 'https':'https://00.11.222.33:4444' }Documentation for theProxyobjectFieldsstring- ipstring- portbool- is_authstring-
usernamestring- passwordProxy.get_dictProxy.get_dict()ParametersnoneReturnsdict- Proxy as a dict in the form{'http':'http://...','https':'https://...'}for use withrequestsExample```py proxy = proxy_manager.next_proxy() proxy_dict =
first_proxy.get_dict() print(proxy_dict.get_dict()) # {
‘http’:’http://00.11.222.33:4444’, ‘https’:’https://00.11.222.33:4444’ }