Package: proxyhub
Porting to Python3.10+ is painful and the progress is moving slowly.We need more volunteers to join. PRs welcome! :joy:ProxyHubProxyHub is an open source tool that asynchronously finds public proxies from multiple sources and concurrently checks them.Features 🌟Finds more than 7000 working proxies from ~50 sources.Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.All proxies are checked to support Cookies and Referer (and POST requests if required).Automatically removes duplicate proxies.Is asynchronous.Requirements 📋Python 3.8+aiohttpaiodnsmaxminddbInstallation 🚀Install locallyTo install last stable release from pypi:$ pip install proxyhubTo install the latest development version from GitHub:$ pip install -U git+https://github.com/ForceFledgling/proxyhub.gitUse pre-built Docker image$ docker pull ForceFledgling/proxyhubBuild bundled one-file executable with pyinstallerRequirementsSupported Operating System: Windows, Linux, MacOSOn UNIX-like systems (Linux / macOSX / BSD)Install these toolsupxobjdump (this tool is usually in the binutils package)$ sudo apt install -y upx-ucl binutils # On Ubuntu / DebianBuildpip install pyinstaller \
&& pip install . \
&& mkdir -p build \
&& cd build \
&& pyinstaller --onefile --name proxyhub --add-data "../proxyhub/data:data" --workpath ./tmp --distpath . --clean ../py2exe_entrypoint.py \
&& rm -rf tmp *.specThe executable is now in the build directoryUsage 💡CLI ExamplesFindFind and show 10 HTTP(S) proxies from United States with the high level of anonymity:$ proxyhub find --types HTTP HTTPS --lvl High --countries US --strict -l 10GrabFind and save to a file 10 US proxies (without a check):$ proxyhub grab --countries US --limit 10 --outfile ./proxies.txtServeRun a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:$ proxyhub serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5Runproxyhub --helpfor more information on the options available.
Runproxyhub <command> --helpfor more information on a command.Basic code exampleFind and show 10 working HTTP(S) proxies:import asyncio
from proxyhub import Broker

async def show(proxies):
    while True:
        proxy = await proxies.get()
        if proxy is None: break
        print('Found proxy: %s' % proxy)

proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
    broker.find(types=['HTTP', 'HTTPS'], limit=10),
    show(proxies))

loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)More examples.Proxy information per requestsHTTPCheckX-Proxy-Infoheader in response.$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://httpbin.org/get
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< X-Proxy-Info: 174.138.42.112:8080
< Date: Mon, 04 May 2020 03:39:40 GMT
< Content-Type: application/json
< Content-Length: 304
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< X-Cache: MISS from ADM-MANAGER
< X-Cache-Lookup: MISS from ADM-MANAGER:880
< Connection: keep-alive
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Cache-Control": "max-age=259200",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-5eaf8e7c-6a1162a1387a1743a49063f4"
  },
  "origin": "...",
  "url": "http://httpbin.org/get"
}
* Connection #0 to host 127.0.0.1 left intactHTTPSWe are not able to modify HTTPS traffic to inject custom header once they start being encrypted. AX-Proxy-Infowill be sent to client afterHTTP/1.1 200 Connection establishedbut not sure how clients can read it.(env) username@host:~/workspace/proxyhub2$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v https://httpbin.org/get
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< X-Proxy-Info: 207.148.22.139:8080
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
...
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5560b2e93580)
> GET /get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Mon, 04 May 2020 03:39:35 GMT
< content-type: application/json
< content-length: 256
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-5eaf8e77-efcb353b0983ad6a90f8bdcd"
  },
  "origin": "...",
  "url": "https://httpbin.org/get"
}
* Connection #0 to host 127.0.0.1 left intactHTTP APIGet info of proxy been used for retrieving specific urlFor HTTP, it's easy.$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/history/url:http://httpbin.org/get
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET http://proxycontrol/api/history/url:http://httpbin.org/get HTTP/1.1
> Host: proxycontrol
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 34
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{"proxy": "..."}For HTTPS, we're not able to know encrypted payload (request), so only hostname can be used.$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/history/url:httpbin.org:443
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET http://proxycontrol/api/history/url:httpbin.org:443 HTTP/1.1
> Host: proxycontrol
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 34
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{"proxy": "..."}
* Connection #0 to host 127.0.0.1 left intactRemove specific proxy from queue$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/remove/PROXY_IP:PROXY_PORT
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET http://proxycontrol/api/remove/... HTTP/1.1
> Host: proxycontrol
> User-Agent: curl/7.58.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 204 No Content
<
* Connection #0 to host 127.0.0.1 left intactTODO 🛠️Check the ping, response time and speed of data transferCheck site access (Google, Twitter, etc) and even your own custom URL'sInformation about uptimeChecksum of data returnedSupport for proxy authenticationFinding outgoing IP for cascading proxyThe ability to specify the address of the proxy without port (try to connect on defaulted ports)Contributing 🤝Fork it:https://github.com/ForceFledgling/proxyhub/forkCreate your feature branch:git checkout -b my-new-featureWe usePoetryto manage dependencies. If need, install dependencies:poetry installCommit your changes:git commit -am 'Add some feature'Push to the branch:git push origin my-new-featureSubmit a pull request!Contributor workflow