Package: proxy.py
Table of ContentsFeaturesInstallUsing PIPStable versionDevelopment versionUsing DockerStable version from Docker HubDevelopment Version from GHCRBuild container locallyUsing HomeBrewStable versionDevelopment versionStart proxy.pyFrom command line when installed using PIPRun itUnderstanding logsEnable DEBUG loggingFrom command line using repo sourceDocker ImageCustomize Startup FlagsPlugin ExamplesHTTP Proxy PluginsShortLink PluginModify Post Data PluginMock Api PluginRedirect To Custom Server PluginFilter By Upstream Host PluginCache Responses PluginMan-In-The-Middle PluginProxy Pool PluginFilter By Client IP PluginModify Chunk Response PluginCloudflare DNS Resolver PluginCustom DNS Resolver PluginCustom Network InterfaceProgram Name PluginHTTP Web Server PluginsWeb Server RouteReverse Proxy PluginsReverse ProxyPlugin OrderingEnd-to-End EncryptionTLS InterceptionTLS Interception With DockerProxy Over SSH TunnelProxy Remote Requests LocallyProxy Local Requests RemotelyEmbed proxy.pyBlocking ModeNon-blocking ModeEphemeral PortLoading PluginsUnit testing with proxy.pyproxy.TestCaseOverride Startup FlagsWithunittest.TestCaseUtilitiesTCPnew_socket_connectionsocket_connectionHttpbuild_http_requestbuild_http_responsePublic Key InfrastructureAPI UsageCLI UsageRun DashboardInspect TrafficChrome DevTools ProtocolFrequently Asked QuestionsDeploying proxy.py in productionWhat not to do?Via RequirementsVia Docker ContainerIntegrate your CI/CD with proxy.pyStable vs DevelopRelease ScheduleThreads vs ThreadlessThreadless Remote vs Local Execution ModeSyntaxError: invalid syntaxUnable to load pluginsUnable to connect with proxy.py from remote hostBasic auth not working with a browserDocker image not working on MacOSValueError: filedescriptor out of range in selectNone:None in access logsOSError when wrapping client for TLS InterceptionPlugin Developer and Contributor GuideHigh level architectureEverything is a pluginInternal DocumentationRead The DocpydocpyreverseDevelopment GuideSetup Local EnvironmentSetup Git HooksSending a Pull RequestProjects Using Proxy.PyBenchmarksFlagsChangelogv2.xv1.xv0.xFeaturesFast & ScalableScale up by using all available cores on the systemThreadless executions using asyncioMade to handletens-of-thousandsconnections / sec#OnMacbookPro2019/2.4GHz8-CoreIntelCorei9/32GBRAM❯ ./helper/benchmark.shCONCURRENCY: 100 workers, TOTAL REQUESTS: 100000 reqSummary:Success rate:	1.0000Total:	2.5489 secsSlowest:	0.0443 secsFastest:	0.0006 secsAverage:	0.0025 secsRequests/sec:	39232.6572Total data:	1.81 MiBSize/request:	19 BSize/sec:	727.95 KiBResponse time histogram:0.001 [5006]  |■■■■■0.001 [19740] |■■■■■■■■■■■■■■■■■■■■■0.002 [29701] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■0.002 [21278] |■■■■■■■■■■■■■■■■■■■■■■0.003 [15376] |■■■■■■■■■■■■■■■■0.004 [6644]  |■■■■■■■0.004 [1609]  |■0.005 [434]   |0.006 [83]    |0.006 [29]    |0.007 [100]   |Latency distribution:10% in 0.0014 secs25% in 0.0018 secs50% in 0.0023 secs75% in 0.0030 secs90% in 0.0036 secs95% in 0.0040 secs99% in 0.0047 secsDetails (average, fastest, slowest):DNS+dialup:	0.0025 secs, 0.0015 secs, 0.0030 secsDNS-lookup:	0.0000 secs, 0.0000 secs, 0.0001 secsStatus code distribution:[200] 100000 responsesConsultThreads vs ThreadlessandThreadless Remote vs Local Execution Modeto control number of CPU cores utilized.SeeBenchmarkfor more details and for how to run benchmarks locally.LightweightUses only~5-20 MBRAMNo memory leaksStart once and forget, no restarts requiredCompressed containers size is only~25 MBNo external dependency other than standard Python libraryProgrammableCustomize proxy behavior usingProxy Server Plugins. Example:--plugins proxy.plugin.ProxyPoolPluginEnable builtinWeb Server. Example:--enable-web-server --plugins proxy.plugin.WebServerPluginEnable builtinReverse Proxy Server. Example:--enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPluginPlugin API is currently indevelopment phase. Expect breaking changes. SeeDeploying proxy.py in productionon how to ensure reliability across code changes.Can listen on multiple portsUse--portsflag to provide additional portsOptionally, use--portflag to override default port8899Capable of serving multiple protocols over the same portReal-time DashboardOptionally, enableproxy.py dashboard.Use--enable-dashboardThen, visithttp://localhost:8899/dashboardInspect, Monitor, Control and Configureproxy.pyat runtimeChrome DevTools ProtocolsupportExtend dashboard frontend usingtypescriptbasedpluginsDashboard is currently indevelopment phaseExpect breaking changes.SecureEnable end-to-end encryption between clients andproxy.pySeeEnd-to-End EncryptionPrivateProtection against DNS based traffic blockersBrowse with malware and adult content protection enabledSeeDNS-over-HTTPSMan-In-The-MiddleCan decrypt TLS traffic between clients and upstream serversSeeTLS InterceptionSupported http protocols for proxy requestshttp(s)http1http1.1with pipelinehttp2websocketsSupport forHAProxy ProtocolSee--enable-proxy-protocolflagStatic file server supportSee--enable-static-serverand--static-server-dirflagsOptimized for large file uploads and downloadsSee--client-recvbuf-size,--server-recvbuf-size,--max-sendbuf-sizeflagsIPv4andIPv6supportSee--hostnameflagUnix domain socket supportSee--unix-socket-pathflagBasic authentication supportSee--basic-authflagPAC (Proxy Auto-configuration) supportSee--pac-fileand--pac-file-url-pathflagsInstallConsultDeploying proxy.py in productionwhen deploying production grade applications usingproxy.py.Using PIPStable Version with PIPInstall fromPyPi❯ pip install --upgrade proxy.pyor from GitHubmasterbranch❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@masterDevelopment Version with PIP❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@developUsing DockerMulti-platform containers are available via:Docker Hublatesttag points to laststablereleasedocker pull abhinavsingh/proxy.py:latestGitHub container registry (GHCR)latesttag points to lastdevelopreleasedocker pull ghcr.io/abhinavsingh/proxy.py:latestStable version container releases are available for following platforms:linux/386linux/amd64linux/arm/v6linux/arm/v7linux/arm64/v8linux/ppc64lelinux/s390xStable Version from Docker HubRunproxy.pylatest container:❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latestDocker daemon will automatically pull the matching platform image.
To run specific target platform container on multi-platform supported servers:❯ docker run -it -p 8899:8899 --rm --platform linux/arm64/v8 abhinavsingh/proxy.py:latestDevelopment Version from GHCRRunproxy.pycontainer from cutting edge code in the develop branch:❯ docker run -it -p 8899:8899 --rm ghcr.io/abhinavsingh/proxy.py:latestBuild Development Version Locally❯ git clone https://github.com/abhinavsingh/proxy.py.git❯ cd proxy.py && make container❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latestdockerimage is currently broken onmacOSdue to incompatibility withvpnkit.Using HomeBrewUpdated formulae forHomeBreware maintained indevelopbranch under thehelper/homebrewdirectory.stableformulae installs the package frommasterbranch.developformulae installs the package fromdevelopbranch.Stable Version with HomeBrew❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/stable/proxy.rbDevelopment Version with HomeBrew❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/develop/proxy.rbStart proxy.pyFrom command line when installed using PIPWhenproxy.pyis installed usingpip,
an executable namedproxyis placed under your$PATH.Run itSimply typeproxyon command line to start with default configuration.❯ proxy...[redacted]... - Loaded plugin proxy.http.proxy.HttpProxyPlugin...[redacted]... - Started 8 threadless workers...[redacted]... - Started 8 acceptors...[redacted]... - Listening on 127.0.0.1:8899Understanding logsThings to notice from above logs:Loaded pluginproxy.pywill loadproxy.http.proxy.HttpProxyPluginby defaultAs name suggests, this core plugin addshttp(s)proxy server capabilities toproxy.pyinstanceStarted N threadless workersBy default,proxy.pywill start as many worker processes as there are CPU cores on the machineUse--num-workersflag to customize number of worker processesSeeThreads vs Threadlessto understand how to control execution modeStarted N acceptorsBy default,proxy.pywill start as many acceptor processes as there are CPU cores on the machineUse--num-acceptorsflag to customize number of acceptor processesSeeHigh Level Architectureto understand relationship between acceptors and workersStarted server on ::1:8899By default,proxy.pylistens on IPv6::1, which is equivalent of IPv4127.0.0.1If you want to accessproxy.pyfrom external host, use--hostname ::or--hostname 0.0.0.0or bind to any other interface available on your machine.SeeCustomNetworkInterfacefor how to customizeproxy.pypublic IP seen by upstream servers.Port 8899Use--portflag to customize default TCP port.Enable DEBUG loggingAll the logs above areINFOlevel logs, default--log-levelforproxy.pyLets startproxy.pywithDEBUGlevel logging:❯ proxy --log-level d...[redacted]... - Open file descriptor soft limit set to 1024...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin...[redacted]... - Started 8 workers...[redacted]... - Started server on ::1:8899You can use single letter to customize log level.  Example:d = DEBUGi = INFOw = WARNINGe = ERRORc = CRITICALAs we can see from the above logs, before starting up:proxy.pytried to set open file limitulimiton the systemDefault value for--open-file-limitused is1024--open-file-limitflag is a no-op onWindowsoperating systemsSeeflagsfor full list of available configuration options.From command line using repo sourceIf you are trying to runproxy.pyfrom source code,
there is no binary file namedproxyin the source code.To startproxy.pyfrom source code follow these instructions:Clone repo❯ git clone https://github.com/abhinavsingh/proxy.py.git❯ cd proxy.pyCreate a Python 3 virtual env❯ python3 -m venv venv❯ source venv/bin/activateInstall deps❯ make lib-depGenerateproxy/common/_scm_version.pyNOTE:Following step is not necessary for editable installs.This file writes SCM detected version toproxy/common/_scm_version.pyfile.❯ ./write-scm-version.shOptionally, run tests❯ makeRunproxy.py❯ python -m proxySeePlugin Developer and Contributor Guideif you plan to work withproxy.pysource code.Docker imageCustomize startup flagsBy defaultdockerbinary is started with IPv4 networking flags:--hostname 0.0.0.0 --port 8899You can override flag from command line when starting the docker container. For example, to checkproxy.pyversion within the docker container, run:❯ docker run -it \
    -p 8899:8899 \
    --rm abhinavsingh/proxy.py:latest \
    -vPlugin ExamplesSeepluginmodule for full code.All the bundled plugin examples also works withhttpstrafficRequire additional flags and certificate generationSeeTLS Interception.Plugin examples are also bundled with Docker image.SeeCustomize startup flagsto try plugins with Docker image.HTTP Proxy PluginsShortLinkPluginAdd support for short links in your favorite browsers / applications.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ShortLinkPluginNow you can speed up your daily browsing experience by visiting your
favorite website using single character domain names :). This works
across all browsers.Following short links are enabled by default:Short LinkDestination URLa/amazon.comi/instagram.coml/linkedin.comf/facebook.comg/google.comt/twitter.comw/web.whatsapp.comy/youtube.comproxy/localhost:8899ModifyPostDataPluginModifies POST request body before sending request to upstream server.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ModifyPostDataPluginBy default plugin replaces POST body content with hard-codedb'{"key": "modified"}'and enforcedContent-Type: application/json.Verify the same usingcurl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post{"args": {},"data": "{\"key\": \"modified\"}","files": {},"form": {},"headers": {"Accept": "*/*","Content-Length": "19","Content-Type": "application/json","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"json": {"key": "modified"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/post"}Note following from the response above:POST data was modified"data": "{\"key\": \"modified\"}".
Originalcurlcommand data was{"key": "value"}.Ourcurlcommand did not add anyContent-Typeheader,
but our plugin did add one"Content-Type": "application/json".
Same can also be verified by looking atjsonfield in the output above:"json": {
 "key": "modified"
},Our plugin also added aContent-Lengthheader to match length
of modified body.MockRestApiPluginMock responses for your server REST API.
Use to test and develop client side applications
without need of an actual upstream REST API server.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ProposedRestApiPluginVerify mock API response usingcurl -x localhost:8899 http://api.example.com/v1/users/{"count": 2, "next": null, "previous": null, "results": [{"email": "you@example.com", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "someone@example.com", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}Verify the same by inspectingproxy.pylogs:... [redacted] ... - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byteAccess log showsNone:Noneas serverip:port.Nonesimply means that
the server connection was never made, since response was returned by our plugin.Now modifyProposedRestApiPluginto returns REST API mock
responses as expected by your clients.RedirectToCustomServerPluginRedirects all incominghttprequests to custom web server.
By default, it redirects client requests to inbuilt web server,
also running on8899port.Startproxy.pyand enable inbuilt web server:❯ proxy \--enable-web-server \--plugins proxy.plugin.RedirectToCustomServerPluginVerify usingcurl -v -x localhost:8899 http://google.com... [redacted] ...
< HTTP/1.1 404 NOT FOUND
< Server: proxy.py v1.0.0
< Connection: Close
<
* Closing connection 0Above404response was returned fromproxy.pyweb server.Verify the same by inspecting the logs forproxy.py.
Along with the proxy request log, you must also see a http web server request log.... [redacted] ... - access_log:1241 - ::1:49525 - GET /
... [redacted] ... - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytesFilterByUpstreamHostPluginDrops traffic by inspecting upstream host.
By default, plugin drops traffic forfacebook.comandwww.facebok.com.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.FilterByUpstreamHostPluginVerify usingcurl -v -x localhost:8899 http://facebook.com:... [redacted] ...< HTTP/1.1 418 I'm a tea pot< Proxy-agent: proxy.py v1.0.0* no chunk, no close, no size. Assume close to signal end<* Closing connection 0Above418 I'm a tea potis sent by our plugin.Verify the same by inspecting logs forproxy.py:... [redacted] ... - handle_readables:1347 - HttpProtocolException type raisedTraceback (most recent call last):... [redacted] ...... [redacted] ... - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytesCacheResponsesPluginCaches Upstream Server Responses.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.CacheResponsesPluginYou may also use the--cache-requestsflag to enable request packet caching for inspection.Verify usingcurl -v -x localhost:8899 http://httpbin.org/get:... [redacted] ...< HTTP/1.1 200 OK< Access-Control-Allow-Credentials: true< Access-Control-Allow-Origin: *< Content-Type: application/json< Date: Wed, 25 Sep 2019 02:24:25 GMT< Referrer-Policy: no-referrer-when-downgrade< Server: nginx< X-Content-Type-Options: nosniff< X-Frame-Options: DENY< X-XSS-Protection: 1; mode=block< Content-Length: 202< Connection: keep-alive<{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/get"}* Connection #0 to host localhost left intactGet path to the cache file fromproxy.pylogs:... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txtVerify contents of the cache filecat /path/to/your/cache/httpbin.org.txtHTTP/1.1 200 OKAccess-Control-Allow-Credentials: trueAccess-Control-Allow-Origin: *Content-Type: application/jsonDate: Wed, 25 Sep 2019 02:24:25 GMTReferrer-Policy: no-referrer-when-downgradeServer: nginxX-Content-Type-Options: nosniffX-Frame-Options: DENYX-XSS-Protection: 1; mode=blockContent-Length: 202Connection: keep-alive{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/get"}ManInTheMiddlePluginModifies upstream server responses.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ManInTheMiddlePluginVerify usingcurl -v -x localhost:8899 http://google.com:... [redacted] ...< HTTP/1.1 200 OK< Content-Length: 28<* Connection #0 to host localhost left intactHello from man in the middleResponse bodyHello from man in the middleis sent by our plugin.ProxyPoolPluginForward incoming proxy requests to a set of upstream proxy servers.Let's start 2 upstream proxies first.  To simulate upstream proxies,
startproxy.pyon port9000and9001❯ proxy --port 9000❯ proxy --port 9001Now, startproxy.pywithProxyPoolPlugin(on default8899port),
pointing to our upstream proxies at9000and9001port.❯ proxy \--plugins proxy.plugin.ProxyPoolPlugin \--proxy-pool localhost:9000 \--proxy-pool localhost:9001Make a curl request via8899proxy:curl -v -x localhost:8899 http://httpbin.org/getVerify that8899proxy forwards requests to upstream proxies
by checking respective logs.If an upstream proxy require credentials, pass them as arguments. Example:--proxy-pool user:pass@upstream.proxy:portFilterByClientIpPluginReject traffic from specific IP addresses. By default this
plugin blocks traffic from127.0.0.1and::1.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.FilterByClientIpPluginSend a request usingcurl -v -x localhost:8899 http://google.com:... [redacted] ...> Proxy-Connection: Keep-Alive>< HTTP/1.1 418 I'm a tea pot< Connection: close<* Closing connection 0Modify plugin to your taste e.g. Allow specific IP addresses only.ModifyChunkResponsePluginThis plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin usesproxy.pycore to parse the chunked encoded response. Then we reconstruct the response using custom hard-coded chunks, ignoring original chunks received from upstream server.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ModifyChunkResponsePluginVerify usingcurl -v -x localhost:8899 http://httpbin.org/stream/5:... [redacted] ...modifychunkresponseplugin* Connection #0 to host localhost left intact* Closing connection 0ModifyModifyChunkResponsePluginto your taste. Example, instead of sending hard-coded chunks, parse and modify the originalJSONchunks received from the upstream server.CloudflareDnsResolverPluginThis plugin usesCloudflarehostedDNS-over-HTTPSAPI(json).DoHmandates a HTTP2 compliant client. Unfortunatelyproxy.pydoes not provide that yet, so we use a dependency. Install it:❯ pip install "httpx[http2]"Now startproxy.pyas:❯ proxy \--plugins proxy.plugin.CloudflareDnsResolverPluginBy default,CloudflareDnsResolverPluginruns insecuritymode and provides malware protection.
Use--cloudflare-dns-mode familyto also enable adult content protection too.CustomDnsResolverPluginThis plugin demonstrate how to use a custom DNS resolution implementation withproxy.py.
This example plugin currently uses Python's in-built resolution mechanism.  Customize code
to your taste.  Example, query your custom DNS server, implementDoHor other mechanisms.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.CustomDnsResolverPluginCustomNetworkInterfaceHttpProxyBasePlugin.resolve_dnscallback can also be used to configurenetwork interfacewhich must be used as thesource_addressfor connection to the upstream server.Seethis threadfor more details.PS: There is no plugin named, butCustomDnsResolverPlugincan be easily customized according to your needs.ProgramNamePluginAttempts to resolve program(application)name for proxy requests originating from the local machine.
If identified, client IP in the access logs is replaced with program name.Startproxy.pyas:❯ proxy \--plugins proxy.plugin.ProgramNamePluginMake a request usingcurl:❯ curl -v -x localhost:8899 https://httpbin.org/getYou must see log lines like this:... [redacted] ... - [I] server.access_log:419 - curl:58096 - CONNECT httpbin.org:443 - 6010 bytes - 1824.62msNoticecurlin-place of::1or127.0.0.1as client IP.IfProgramNamePlugindoes not work reliably on your operating system, kindly contribute by sending a pull request and/or open an issue.  Thank you!!!HTTP Web Server PluginsWeb Server RouteDemonstrates inbuilt web server routing using plugin.Startproxy.pyas:❯ proxy --enable-web-server \--plugins proxy.plugin.WebServerPluginVerify usingcurl -v localhost:8899/http-route-example, should return:HTTP route responseReverse Proxy PluginsExtends in-built Web Server to add Reverse Proxy capabilities.Reverse ProxyStartproxy.pyas:❯ proxy --enable-reverse-proxy \--plugins proxy.plugin.ReverseProxyPluginWith default configuration,ReverseProxyPluginplugin is equivalent to
followingNginxconfig:location /get {proxy_pass http://httpbin.org/get}Verify usingcurl -v localhost:8899/get:{"args": {},"headers": {"Accept": "*/*","Host": "localhost","User-Agent": "curl/7.64.1"},"origin": "1.2.3.4, 5.6.7.8","url": "https://localhost/get"}Plugin OrderingWhen using multiple plugins, depending upon plugin functionality,
it might be worth considering the order in which plugins are passed
on the command line.Plugins are called in the same order as they are passed. Example,
say we are using bothFilterByUpstreamHostPluginandRedirectToCustomServerPlugin. Idea is to drop all incominghttprequests forfacebook.comandwww.facebook.comand redirect otherhttprequests to our inbuilt web server.Hence, in this scenario it is important to useFilterByUpstreamHostPluginbeforeRedirectToCustomServerPlugin.
If we enableRedirectToCustomServerPluginbeforeFilterByUpstreamHostPlugin,facebookrequests will also get redirected to inbuilt web server,
instead of being dropped.End-to-End EncryptionBy default,proxy.pyuseshttpprotocol for communication with clients e.g.curl,browser. For enabling end-to-end encrypting usingtls/httpsfirst generate certificates.Checkoutthe repository and run:make https-certificatesStartproxy.pyas:❯ proxy \--cert-file https-cert.pem \--key-file https-key.pemVerify usingcurl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get:{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/get"}If you want to avoid passing--proxy-cacertflag, also consider signing generated SSL certificates. Example:First, generate CA certificates:make ca-certificatesThen, sign SSL certificate:make sign-https-certificatesNow restart the server with--cert-file https-signed-cert.pemflag. Note that you must also trust generatedca-cert.pemin your system keychain.TLS InterceptionBy default,proxy.pywill not decrypthttpstraffic between client and server.
To enable TLS interception first generate root CA certificates:❯ make ca-certificatesLets also enableCacheResponsePluginso that we can verify decrypted
response from the server. Startproxy.pyas:❯ proxy \--plugins proxy.plugin.CacheResponsesPlugin \--ca-key-file ca-key.pem \--ca-cert-file ca-cert.pem \--ca-signing-key-file ca-signing-key.pemAlso provide explicit CA bundle path needed for validation of peer certificates. See--ca-fileflag.Verify TLS interception usingcurl❯ curl -v -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get*  issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; emailAddress=proxyca@mailserver.com*  SSL certificate verify ok.> GET /get HTTP/1.1... [redacted] ...< Connection: keep-alive<{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/get"}Theissuerline confirms that response was intercepted.Also verify the contents of cached response file. Get path to the cache
file fromproxy.pylogs.❯ cat /path/to/your/tmp/directory/httpbin.org-1569452863.924174.txtHTTP/1.1 200 OKAccess-Control-Allow-Credentials: trueAccess-Control-Allow-Origin: *Content-Type: application/jsonDate: Wed, 25 Sep 2019 23:07:05 GMTReferrer-Policy: no-referrer-when-downgradeServer: nginxX-Content-Type-Options: nosniffX-Frame-Options: DENYX-XSS-Protection: 1; mode=blockContent-Length: 202Connection: keep-alive{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "1.2.3.4, 5.6.7.8","url": "https://httpbin.org/get"}Viola!!! If you remove CA flags, encrypted data will be found in the
cached file instead of plain text.Now use CA flags with otherplugin examplesto see them work withhttpstraffic.TLS Interception With DockerImportant notes about TLS Interception with Docker container:Sincev2.2.0,proxy.pydocker container also ships withopenssl. This allowsproxy.pyto generate certificates on the fly for TLS Interception.For security reasons,proxy.pydocker container does not ship with
CA certificates.Here is how to start aproxy.pydocker container
with TLS Interception:Generate CA certificates on host computer❯ make ca-certificatesCopy all generated certificates into a separate directory. We'll later mount this directory into our docker container❯ mkdir /tmp/ca-certificates❯ cp ca-cert.pem ca-key.pem ca-signing-key.pem /tmp/ca-certificatesStart docker container❯ docker run -it --rm \-v /tmp/ca-certificates:/tmp/ca-certificates \-p 8899:8899 \abhinavsingh/proxy.py:latest \--hostname 0.0.0.0 \--plugins proxy.plugin.CacheResponsesPlugin \--ca-key-file /tmp/ca-certificates/ca-key.pem \--ca-cert-file /tmp/ca-certificates/ca-cert.pem \--ca-signing-key /tmp/ca-certificates/ca-signing-key.pem-v /tmp/ca-certificates:/tmp/ca-certificatesflag mounts our CA certificate directory in container environment--plugins proxy.plugin.CacheResponsesPluginenablesCacheResponsesPluginso that we can inspect intercepted traffic--ca-*flags enable TLS Interception.From another terminal, try TLS Interception usingcurl. You can omit--cacertflag if CA certificate is already trusted by the system.❯ curl -v \--cacert ca-cert.pem \-x 127.0.0.1:8899 \https://httpbin.org/getVerifyissuerfield from response headers.* Server certificate:*  subject: CN=httpbin.org; C=NA; ST=Unavailable; L=Unavailable; O=Unavailable; OU=Unavailable*  start date: Jun 17 09:26:57 2020 GMT*  expire date: Jun 17 09:26:57 2022 GMT*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"*  issuer: CN=example.com*  SSL certificate verify ok.Back on docker terminal, copy response dump path logs....[redacted]... [I] access_log:338 - 172.17.0.1:56498 - CONNECT httpbin.org:443 - 1031 bytes - 1216.70 ms...[redacted]... [I] close:49 - Cached response at /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txtIn another terminal,catthe response dump:❯ docker exec -it $(docker ps | grep proxy.py | awk '{ print $1 }') cat /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txtHTTP/1.1 200 OK...[redacted]...{...[redacted]...,"url": "http://httpbin.org/get"}Proxy Over SSH TunnelThis is a WIP and may not work as documentedRequiresparamikoto work.Seerequirements-tunnel.txtProxy Remote Requests Locally|
+------------+          |            +----------+
|   LOCAL    |          |            |  REMOTE  |
|   HOST     | <== SSH ==== :8900 == |  SERVER  |
+------------+          |            +----------+
:8899 proxy.py          |
                        |
                     FIREWALL
                  (allow tcp/22)WhatProxy HTTP(s) requests made on aremoteserver throughproxy.pyserver
running onlocalhost.HowRequestedremoteport is forwarded over the SSH connection.proxy.pyrunning on thelocalhosthandles and responds toremoteproxy requests.RequirementslocalhostMUST have SSH access to theremoteserverremoteserver MUST be configured to proxy HTTP(s) requests
through the forwarded port number e.g.:8900.remoteandlocalhostports CAN be same e.g.:8899.:8900is chosen in ascii art for differentiation purposes.Try itStartproxy.pyas:❯ # On localhost❯ proxy --enable-tunnel \--tunnel-username username \--tunnel-hostname ip.address.or.domain.name \--tunnel-port 22 \--tunnel-remote-port 8899 \--tunnel-ssh-key /path/to/ssh/private.key \--tunnel-ssh-key-passphrase XXXXX...[redacted]... [I] listener.setup:97 - Listening on 127.0.0.1:8899...[redacted]... [I] pool.setup:106 - Started 16 acceptors in threadless (local) mode...[redacted]... [I] transport._log:1873 - Connected (version 2.0, client OpenSSH_7.6p1)...[redacted]... [I] transport._log:1873 - Authentication (publickey) successful!...[redacted]... [I] listener.setup:116 - SSH connection established to ip.address.or.domain.name:22......[redacted]... [I] listener.start_port_forward:91 - :8899 forwarding successful...Make a HTTP proxy request onremoteserver and
verify that response contains public IP address oflocalhostas origin:❯ # On remote❯ curl -x 127.0.0.1:8899 http://httpbin.org/get{"args": {},"headers": {"Accept": "*/*","Host": "httpbin.org","User-Agent": "curl/7.54.0"},"origin": "x.x.x.x, y.y.y.y","url": "https://httpbin.org/get"}Also, verify thatproxy.pylogs onlocalhostcontainsremoteIP as client IP.access_log:328 - remote:52067 - GET httpbin.org:80Proxy Local Requests Remotely|
+------------+          |     +----------+
|   LOCAL    |          |     |  REMOTE  |
|   HOST     | === SSH =====> |  SERVER  |
+------------+          |     +----------+
                        |     :8899 proxy.py
                        |
                    FIREWALL
                 (allow tcp/22)Not planned.If you have a valid use case, kindly open an issue.  You are always welcome to send
contributions via pull-requests to add this functionality :)To proxy local requests remotely, make use ofProxy Pool Plugin.Embed proxy.pyBlocking ModeStartproxy.pyin embedded mode with default configuration
by usingproxy.mainmethod. Example:importproxyif__name__=='__main__':proxy.main()Customize startup flags by passing them as kwargs:importipaddressimportproxyif__name__=='__main__':proxy.main(hostname=ipaddress.IPv6Address('::1'),port=8899)Note that:mainis equivalent to startingproxy.pyfrom command line.maindoes not accept anyargs(onlykwargs).mainwill automatically consume any availablesys.argvasargs.mainwill block untilproxy.pyshuts down.Non-blocking ModeStartproxy.pyin non-blocking embedded mode with default configuration
by usingProxycontext manager: Example:importproxyif__name__=='__main__':withproxy.Proxy()asp:# Uncomment the line below and# implement your app your logic hereproxy.sleep_loop()Note that:Proxyis similar tomain, exceptProxywill not block.Internally,Proxyis a context manager which will startproxy.pywhen called and will shut it down once the scope ends.Unlikemain, startup flags withProxycan also be customized
by usingargsandkwargs. e.g.Proxy(['--port', '8899'])or
by using passing flags as kwargs e.g.Proxy(port=8899).Unlikemain,Proxywill not inspectsys.argv.Ephemeral PortUse--port=0to bindproxy.pyon a random port allocated by the kernel.In embedded mode, you can access this port.  Example:importproxyif__name__=='__main__':withproxy.Proxy()asp:print(p.flags.port)proxy.sleep_loop()flags.portwill give you access to the random port allocated by the kernel.Loading PluginsUsers can use--pluginsflag multiple times to load multiple plugins.
SeeUnable to load pluginsif you are running into issues.When using in embedded mode, you have a few more options.  Example:Provide a fully-qualified name of the plugin class asbytesto theproxy.mainmethod orproxy.Proxycontext manager.Providetypeinstance of the plugin class. This is especially useful if you plan to define plugins at runtime.Example, load a single plugin using--pluginsflag:importproxyif__name__=='__main__':proxy.main(plugins=['proxy.plugin.CacheResponsesPlugin'])For simplicity, you can also pass the list of plugins as a keyword argument toproxy.mainor theProxyconstructor.Example:importproxyfromproxy.pluginimportFilterByUpstreamHostPluginif__name__=='__main__':proxy.main(plugins=[b'proxy.plugin.CacheResponsesPlugin',FilterByUpstreamHostPlugin,])Unit testing with proxy.pyproxy.TestCaseTo setup and tear downproxy.pyfor your Pythonunittestclasses, simply useproxy.TestCaseinstead ofunittest.TestCase.
Example:importproxyclassTestProxyPyEmbedded(proxy.TestCase):deftest_my_application_with_proxy(self)->None:self.assertTrue(True)Note that:proxy.TestCaseoverridesunittest.TestCase.run()method to setup and tear downproxy.py.proxy.pyserver will listen on a random available port on the system.
This random port is available asself.PROXY.flags.portwithin your test cases.Only a single acceptor and worker is started by default (--num-workers 1 --num-acceptors 1) for faster setup and tear down.Most importantly,proxy.TestCasealso ensuresproxy.pyserver
is up and running before proceeding with execution of tests. By default,proxy.TestCasewill wait for10 secondsforproxy.pyserver to start,
upon failure aTimeoutErrorexception will be raised.Override startup flagsTo override default startup flags, define aPROXY_PY_STARTUP_FLAGSvariable in your test class.
Example:classTestProxyPyEmbedded(TestCase):PROXY_PY_STARTUP_FLAGS=['--num-workers','2','--num-acceptors','1','--enable-web-server',]deftest_my_application_with_proxy(self)->None:self.assertTrue(True)Seetest_embed.pyfor full working example.Withunittest.TestCaseIf for some reasons you are unable to directly useproxy.TestCase,
then simply overrideunittest.TestCase.runyourself to setup and tear downproxy.py.
Example:importunittestimportproxyclassTestProxyPyEmbedded(unittest.TestCase):deftest_my_application_with_proxy(self)->None:self.assertTrue(True)defrun(self,result:Optional[unittest.TestResult]=None)->Any:withproxy.start(['--num-workers','1','--num-acceptors','1','--port','... random port ...']):super().run(result)or simply setup / tear downproxy.pywithinsetUpClassandteardownClassclass methods.UtilitiesTCP Socketsnew_socket_connectionAttempts to create an IPv4 connection, then IPv6 and
finally a dual stack connection to provided address.>>>conn=new_socket_connection(('httpbin.org',80))>>>...[useconnection]...>>>conn.close()socket_connectionsocket_connectionis a convenient decorator + context manager
aroundnew_socket_connectionwhich ensuresconn.closeis implicit.As a context manager:>>>withsocket_connection(('httpbin.org',80))asconn:>>>...[useconnection]...As a decorator:>>>@socket_connection(('httpbin.org',80))>>>defmy_api_call(conn,*args,**kwargs):>>>...[useconnection]...HTTP Clientbuild_http_requestGenerate HTTP GET request>>>build_http_request(b'GET',b'/')b'GET / HTTP/1.1\r\n\r\n'Generate HTTP GET request with headers>>>build_http_request(b'GET',b'/',conn_close=True)b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'Generate HTTP POST request with headers and body>>>importjson>>>build_http_request(b'POST',b'/form',headers={b'Content-type':b'application/json'},body=proxy.bytes_(json.dumps({'email':'hello@world.com'})))b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'build_http_responsebuild_http_response(status_code:int,protocol_version:bytes=HTTP_1_1,reason:Optional[bytes]=None,headers:Optional[Dict[bytes,bytes]]=None,body:Optional[bytes]=None)->bytesPKIAPI Usagegen_private_keygen_private_key(key_path:str,password:str,bits:int=2048,timeout:int=10)->boolgen_public_keygen_public_key(public_key_path:str,private_key_path:str,private_key_password:str,subject:str,alt_subj_names:Optional[List[str]]=None,extended_key_usage:Optional[str]=None,validity_in_days:int=365,timeout:int=10)->boolremove_passphraseremove_passphrase(key_in_path:str,password:str,key_out_path:str,timeout:int=10)->boolgen_csrgen_csr(csr_path:str,key_path:str,password:str,crt_path:str,timeout:int=10)->boolsign_csrsign_csr(csr_path:str,crt_path:str,ca_key_path:str,ca_key_password:str,ca_crt_path:str,serial:str,alt_subj_names:Optional[List[str]]=None,extended_key_usage:Optional[str]=None,validity_in_days:int=365,timeout:int=10)->boolSeepki.pyandtest_pki.pyfor usage examples.CLI UsageUseproxy.common.pkimodule for:Generation of public and private keysGenerating CSR requestsSigning CSR requests using custom CA.python -m proxy.common.pki -husage: pki.py [-h] [--password PASSWORD] [--private-key-path PRIVATE_KEY_PATH][--public-key-path PUBLIC_KEY_PATH] [--subject SUBJECT]actionproxy.py v2.2.0 : PKI Utilitypositional arguments:action                Valid actions: remove_passphrase, gen_private_key,gen_public_key, gen_csr, sign_csroptional arguments:-h, --help            show this help message and exit--password PASSWORD   Password to use for encryption. Default: proxy.py--private-key-path PRIVATE_KEY_PATHPrivate key path--public-key-path PUBLIC_KEY_PATHPublic key path--subject SUBJECT     Subject to use for public key generation. Default:/CN=example.comInternal DocumentationRead The DocVisitproxypy.readthedocs.ioBuild locally using:make lib-docpydocCode is well documented.  Grab the source code and run:pydoc3 proxypyreverseGenerate class level hierarchy UML diagrams for in-depth analysis:make lib-pyreverseRun DashboardDashboard is currently under development and not yet bundled withpippackages.
To run dashboard, you must checkout the source.Dashboard is written in Typescript and SCSS, so let's build it first using:❯ make dashboardAlso build the embeddedChrome DevToolsif you plan on using it:❯ make devtoolsNow startproxy.pywith dashboard plugin and by overriding root directory for static server:❯ proxy --enable-dashboard --static-server-dir dashboard/public...[redacted]... - Loaded plugin proxy.http.server.HttpWebServerPlugin...[redacted]... - Loaded plugin proxy.dashboard.dashboard.ProxyDashboard...[redacted]... - Loaded plugin proxy.dashboard.inspect_traffic.InspectTrafficPlugin...[redacted]... - Loaded plugin proxy.http.inspector.DevtoolsProtocolPlugin...[redacted]... - Loaded plugin proxy.http.proxy.HttpProxyPlugin...[redacted]... - Listening on ::1:8899...[redacted]... - Core Event enabledCurrently, enabling dashboard will also enable all the dashboard plugins.Visit dashboard:❯ open http://localhost:8899/dashboard/Inspect TrafficThis is a WIP and may not work as documentedWait for embeddedChrome Dev Consoleto load.  Currently, detail about all traffic flowing
throughproxy.pyis pushed to theInspect Traffictab.  However, received payloads are not
yet integrated with the embedded developer console.Current functionality can be verified by opening theDev Consoleof dashboard and inspecting
the websocket connection that dashboard established with theproxy.pyserver.Chrome DevTools ProtocolFor scenarios where you want direct access toChrome DevToolsprotocol websocket endpoint,
startproxy.pyas:❯ proxy --enable-devtools --enable-eventsNow point your CDT instance tows://localhost:8899/devtools.Frequently Asked QuestionsDeploying proxy.py in productionListed below are a few strategies for usingproxy.pyin your private/production/corporate projects.What not to do?You MUSTavoid forkingthe repository"just"to put your plugin code inproxy/plugindirectory.  Forking is recommended workflow for project contributors, NOT for project users.Instead, use one of the suggested approaches from below.Then load your plugins using--plugin,--pluginsflags orpluginkwargs.Seeskeletonapp for example standalone project usingproxy.py.Via RequirementsIt ishighlyrecommended that you useproxy.pyviarequirements.txtor similar dependency management setups. This will allow you to take advantages of regular performance updates, bug fixes, security patches and other improvements happening in theproxy.pyecosystem.  Example:Use--preoption to depend upon lastpre-release❯ pip install proxy.py --prePre-releases are similar to depending upondevelopbranch code, just that pre-releases may not point to theHEAD.  This could happen because pre-releases are NOT made available onPyPiafter every PR merge.UseTestPyPiwith--preoption to depend upondevelopbranch code❯ pip install -i https://test.pypi.org/simple/ proxy.py --preA pre-release is made available onTestPyPiafter every PR merge.Use laststablerelease codeAs usual, simply use:❯ pip install proxy.pyVia Docker ContainerIf you are into deploying containers, then simply build your image from baseproxy.pycontainer images.UseGHCRto build fromdevelopbranch code:FROM ghcr.io/abhinavsingh/proxy.py:latest as basePS: I use GHCR latest for several production level projectsUseDockerHubto build from laststablerelease code:FROM abhinavsingh/proxy.py:latest as basePS: IMHO, container based strategy isthe best approachand the only strategy thatI use myself.Integrate your CI/CD with proxy.pyHey, but you keep making breaking changes in the develop branch.I hear you.  And hence, for your production grade applications, youMUSTintegrate application CI/CD withproxy.py.  You must make sure that your application builds and passes its tests for every PR merge into theproxy.pyupstream repo.If your application repository is public, in certain scenarios, PR authors may send patch PRs for all dependents to maintain backward incompatibility and green CI/CD.CI/CD integration ensure your app continues to build with latestproxy.pycode.  Depending upon where you host your code, use the strategy listed below:GitHubTBDGoogle Cloud BuildTBDAWSTBDAzureTBDOthersTBDAt some stage, we'll deprecatemasterbranch segregation and simply maintain adevelopbranch.  As dependents can maintain stability via CI/CD integrations. Currently, it's hard for a production grade project to blindly depend upondevelopbranch.Stable vs Developmasterbranch contains lateststablecode and is available viaPyPirepository andDockercontainers viadocker.ioandghcr.ioregistries.Issues reported forstablereleases are considered with top-priority.  However, currently we don't back port fixes into older releases.  Example, if you reported an issue inv2.3.1, but currentmasterbranch now containsv2.4.0rc1.  Then, the fix will land inv2.4.0rc2.developbranch contains cutting edge changesDevelopment branch is kept stable(most of the times).But, if you want100% reliabilityand serving users inproduction environment, ALWAYS use the stable version.Release ScheduleAvX.Y.ZrcNpull request is created once a month which mergesdevelop→master.  Find below how code flows from a pull request to the next stable release.Development release is deployed fromdevelop→test.pypi.orgafter every pull request mergeAlpha release is deployed fromdevelop→pypi.orgbeforemerging thevX.Y.Z.rcNpull request fromdevelop→masterbranch.  There can be multiple alpha releases made before merging thercpull requestBeta release is deployed frommaster→pypi.org.  Beta releases are made in preparation ofrcreleases and can be skipped if unnecessaryRelease candidate is deployed frommaster→pypi.org.  Release candidates are always made available before final stable releaseStable release is deployed frommaster→pypi.orgThreads vs Threadlessv1.xproxy.pyused to spawn new threads for handling client requests.v2.0+proxy.pyadded support for threadless execution of client requests usingasyncio.v2.4.0+Threadless execution was turned ON by default forPython 3.8+onmacandlinuxenvironments.proxy.pythreadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using--threadedflag.ForwindowsandPython < 3.8, you can still try out threadless mode by startingproxy.pywith--threadlessflag.If threadless works for you, consider sending a PR by editing_env_threadless_compliantmethod in theproxy/common/constants.pyfile.Threadless Remote vs Local execution modeOriginal threadless implementation usedremoteexecution mode.  This is also depicted underHigh level architectureas ASCII art.Underremoteexecution mode, acceptors delegate incoming client connection processing to a remote worker process.  By default, acceptors delegate connections in round-robin fashion.  Worker processing the request may or may not be running on the same CPU core as the acceptor.  This architecture scales well for high throughput, but results in spawning two process per CPU core.Example, if there are N-CPUs on the machine, by default, N acceptors and N worker processes are started.  You can tune number of processes using--num-acceptorsand--num-workersflag.  You might want more workers than acceptors or vice versa depending upon your use case.In v2.4.x,localexecution mode was added, mainly to reduce number of processes spawned by default.  This model serves well for day-to-day single user use cases and for developer testing scenarios.  Underlocalexecution mode, acceptors delegate client connections to a companion thread, instead of a remote process.localexecution mode ensure CPU affinity, unlike in theremotemode where acceptor and worker might be running on different CPU cores.--local-executor 1was made default in v2.4.x series.  Underlocalexecution mode,--num-workersflag has no effect, as no remote workers are started.To useremoteexecution mode, use--local-executor 0flag.  Then use--num-workersto tune number of worker processes.SyntaxError: invalid syntaxproxy.pyis strictly typed and uses Pythontypingannotations. Example:>>>my_strings:List[str]=[]>>>#############^^^^^^^^^#####Hence a Python version that understands typing annotations is required.
Make sure you are usingPython 3.6+.Verify the version before runningproxy.py:❯ python --versionAlltypingannotations can be replaced withcomment-onlyannotations. Example:>>>my_strings=[]# List[str]>>>################^^^^^^^^^^^It will enableproxy.pyto run on Pythonpre-3.6, even on2.7.
However, as all future versions of Python will supporttypingannotations,
this has not been considered.Unable to load pluginsMake sure plugin modules are discoverable by adding them toPYTHONPATH. Example:PYTHONPATH=/path/to/my/app proxy --plugins my_app.proxyPlugin...[redacted]... - Loaded plugin proxy.HttpProxyPlugin...[redacted]... - Loaded plugin my_app.proxyPluginOR, simply pass fully-qualified path as parameter, e.g.proxy --plugins /path/to/my/app/my_app.proxyPluginHere is a quick working example:Contents of/tmp/plugfolder╰─ ls -1 /tmp/plug                                                                                                                       ─╯my_plugin.pyCustomMyPluginclass╰─ cat /tmp/plug/my_plugin.py                                                                                                            ─╯from proxy.http.proxy import HttpProxyBasePluginclass MyPlugin(HttpProxyBasePlugin):passThis is an empty plugin for demonstrating external plugin usage. You must implement necessary methods to make your plugins work for real trafficStartproxy.pywithMyPlugin╰─ PYTHONPATH=/tmp/plug proxy --plugin my_plugin.MyPlugin                                                                      ─╯...[redacted]... - Loaded plugin proxy.http.proxy.HttpProxyPlugin...[redacted]... - Loaded plugin my_plugin.MyPlugin...[redacted]... - Listening on ::1:8899Unable to connect with proxy.py from remote hostMake sureproxy.pyis listening on correct network interface.
Try following flags:For IPv6--hostname ::For IPv4--hostname 0.0.0.0Basic auth not working with a browserMost likely it's a browser integration issue with system keychain.First verify that basic auth is working usingcurlcurl -v -x username:password@localhost:8899 https://httpbin.org/getSeethis threadfor further details.Docker image not working on macOSIt's a compatibility issue withvpnkit.Seemoby/vpnkit exhausts docker resourcesandConnection refused: The proxy could not connectfor some background.GCE log viewer integration for proxy.pyA starterfluentd.conftemplate is available.Copy this configuration file asproxy.py.confunder/etc/google-fluentd/config.d/Updatepathfield to log file path as used with--log-fileflag.
By default/tmp/proxy.logpath is tailed.Reloadgoogle-fluentd:sudo service google-fluentd restartNowproxy.pylogs can be browsed usingGCE log viewer.ValueError: filedescriptor out of range in selectproxy.pyis made to handle thousands of connections per second
without any socket leaks.Make use of--open-file-limitflag to customizeulimit -n.Make sure to adjust--backlogflag for higher concurrency.If nothing helps,open an issuewithrequests per secondsent and output of following debug script:❯ ./helper/monitor_open_files.sh <proxy-py-pid>None:None in access logsSometimes you may seeNone:Nonein access logs. It simply means
that an upstream server connection was never established i.e.upstream_host=None,upstream_port=None.There can be several reasons for no upstream connection,
few obvious ones include:Client established a connection but never completed the request.A plugin returned a response prematurely, avoiding connection to upstream server.OSError when wrapping client for TLS InterceptionWithTLS Interceptionon, you might occasionally see following exceptions:2021-11-06 23:33:34,540 - pid:91032 [E] server.intercept:678 - OSError when wrapping clientTraceback (most recent call last):...[redacted]......[redacted]......[redacted]...ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:997)...[redacted]... - CONNECT oauth2.googleapis.com:443 - 0 bytes - 272.08 msSome clients can throwTLSV1_ALERT_UNKNOWN_CAif they cannot verify the certificate of the server
because it is signed by an unknown issuer CA.  Which is the case when we are doing TLS interception.
This can be for a variety of reasons e.g. certificate pinning etc.Another exception you might see isCERTIFICATE_VERIFY_FAILED:2021-11-06 23:36:02,002 - pid:91033 [E] handler.handle_readables:293 - Exception while receiving from client connection <socket.socket fd=28, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 51961)> with reason SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')Traceback (most recent call last):...[redacted]......[redacted]......[redacted]...ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)...[redacted]... - CONNECT init.push.apple.com:443 - 0 bytes - 892.99 msIn future, we might support serving original HTTPS content for such clients while still
performing TLS interception in the background.  This will keep the clients happy without
impacting our ability to TLS intercept.  Unfortunately, this feature is currently not available.Another example withSSLEOFErrorexception:2021-11-06 23:46:40,446 - pid:91034 [E] server.intercept:678 - OSError when wrapping clientTraceback (most recent call last):...[redacted]......[redacted]......[redacted]...ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:997)...[redacted]... - CONNECT stock.adobe.io:443 - 0 bytes - 685.32 msPlugin Developer and Contributor GuideHigh level architecture+-------------+|             ||  Proxy([])  ||             |+------+------+||+-----------v--------------+|                          ||    AcceptorPool(...)     ||                          |+------------+-------------+|+-----------------+             |           +-----------------+|                 |             |           |                 ||   Acceptor(..)  <-------------+----------->  Acceptor(..)   ||                 |                         |                 |+---+-------------+                         +---------+-------+|                                                 ||                                                 ||    +------++------++------++------++------+     ||    |      ||      ||      ||      ||      |     |+---->      ||      ||      ||      ||      <-----+|      ||      ||      ||      ||      |+------++------++------++------++------+Threadless Worker Processesproxy.pyis made with performance in mind.  By default,proxy.pywill try to utilize all available CPU cores to it for accepting new
client connections. This is achieved by startingAcceptorPoolwhich
listens on configured server port. Then,AcceptorPoolstartsAcceptorprocesses (--num-acceptors) to accept incoming client connections.
Alongside, if--threadlessis enabled,ThreadlessPoolis setup
which startsThreadlessprocesses (--num-workers) to handle
the incoming client connections.EachAcceptorprocess delegates the accepted client connection
to a threadless process viaWorkclass.  Currently,HttpProtocolHandleris the default work class.HttpProtocolHandlersimply assumes that incoming clients will follow
HTTP specification.  Specific HTTP proxy and HTTP server implementations
are written as plugins ofHttpProtocolHandler.See documentation ofHttpProtocolHandlerPluginfor available lifecycle hooks.
UseHttpProtocolHandlerPluginto add new features for http(s) clients. Example,
SeeHttpWebServerPlugin.Everything is a pluginWithinproxy.pyeverything is a plugin.We enabledproxy serverplugins using--pluginsflag.
Proxy serverHttpProxyPluginis a plugin ofHttpProtocolHandler.
Further, Proxy server allows plugin throughHttpProxyBasePluginspecification.All the proxy serverplugin exampleswere implementingHttpProxyBasePlugin. See documentation ofHttpProxyBasePluginfor available
lifecycle hooks. UseHttpProxyBasePluginto modify behavior of http(s) proxy protocol
between client and upstream server. Example,FilterByUpstreamHostPlugin.We also enabled inbuiltweb serverusing--enable-web-server.
Web serverHttpWebServerPluginis a plugin ofHttpProtocolHandlerand implementsHttpProtocolHandlerPluginspecification.There also is a--disable-http-proxyflag. It disables inbuilt proxy server.
Use this flag with--enable-web-serverflag to runproxy.pyas a programmable
http(s) server.Development GuideSetup Local EnvironmentContributors must startproxy.pyfrom source to verify and develop new features / fixes.SeeRun proxy.py from command line using repo sourcefor details.OnmacOSyou must installPythonusingpyenv, asPythoninstalled viahomebrewtends
to be problematic.  See linked thread for more details.Setup Git HooksPre-commit hook ensures tests are passing.cd /path/to/proxy.pyln -s $(PWD)/git-pre-commit .git/hooks/pre-commitPre-push hook ensures lint and tests are passing.cd /path/to/proxy.pyln -s $(PWD)/git-pre-push .git/hooks/pre-pushSending a Pull RequestEvery pull request is tested using GitHub actions.SeeGitHub workflowfor list of tests.Projects Using Proxy.PySome of the projects usingproxy.pyray-projectaio-libswifipumpkin3MerossIotpyshortenersSlack APIibeamPyPaperBotFor full list seeused byBenchmarksSeeBenchmarkdirectory on how to run benchmark comparisons with other OSS web servers.To run standalone benchmark forproxy.py, use the following command from repo root:❯ ./helper/benchmark.shFlags❯ proxy -husage: -m [-h] [--tunnel-hostname TUNNEL_HOSTNAME] [--tunnel-port TUNNEL_PORT][--tunnel-username TUNNEL_USERNAME][--tunnel-ssh-key TUNNEL_SSH_KEY][--tunnel-ssh-key-passphrase TUNNEL_SSH_KEY_PASSPHRASE][--tunnel-remote-port TUNNEL_REMOTE_PORT] [--threadless][--threaded] [--num-workers NUM_WORKERS] [--enable-events][--local-executor LOCAL_EXECUTOR] [--backlog BACKLOG][--hostname HOSTNAME] [--port PORT] [--ports PORTS [PORTS ...]][--port-file PORT_FILE] [--unix-socket-path UNIX_SOCKET_PATH][--num-acceptors NUM_ACCEPTORS] [--version] [--log-level LOG_LEVEL][--log-file LOG_FILE] [--log-format LOG_FORMAT][--open-file-limit OPEN_FILE_LIMIT][--plugins PLUGINS [PLUGINS ...]] [--enable-dashboard][--basic-auth BASIC_AUTH] [--enable-ssh-tunnel][--work-klass WORK_KLASS] [--pid-file PID_FILE][--enable-proxy-protocol] [--enable-conn-pool] [--key-file KEY_FILE][--cert-file CERT_FILE] [--client-recvbuf-size CLIENT_RECVBUF_SIZE][--server-recvbuf-size SERVER_RECVBUF_SIZE][--max-sendbuf-size MAX_SENDBUF_SIZE] [--timeout TIMEOUT][--disable-http-proxy] [--disable-headers DISABLE_HEADERS][--ca-key-file CA_KEY_FILE] [--ca-cert-dir CA_CERT_DIR][--ca-cert-file CA_CERT_FILE] [--ca-file CA_FILE][--ca-signing-key-file CA_SIGNING_KEY_FILE][--auth-plugin AUTH_PLUGIN] [--cache-requests][--cache-by-content-type] [--cache-dir CACHE_DIR][--proxy-pool PROXY_POOL] [--enable-web-server][--enable-static-server] [--static-server-dir STATIC_SERVER_DIR][--min-compression-length MIN_COMPRESSION_LENGTH][--enable-reverse-proxy] [--pac-file PAC_FILE][--pac-file-url-path PAC_FILE_URL_PATH][--cloudflare-dns-mode CLOUDFLARE_DNS_MODE][--filtered-upstream-hosts FILTERED_UPSTREAM_HOSTS][--filtered-client-ips-mode FILTERED_CLIENT_IPS_MODE][--filtered-client-ips FILTERED_CLIENT_IPS][--filtered-url-regex-config FILTERED_URL_REGEX_CONFIG]proxy.py v2.4.3.dev14+gc6b2de6.d20220605optional arguments:-h, --help            show this help message and exit--tunnel-hostname TUNNEL_HOSTNAMEDefault: None. Remote hostname or IP address to whichSSH tunnel will be established.--tunnel-port TUNNEL_PORTDefault: 22. SSH port of the remote host.--tunnel-username TUNNEL_USERNAMEDefault: None. Username to use for establishing SSHtunnel.--tunnel-ssh-key TUNNEL_SSH_KEYDefault: None. Private key path in pem format--tunnel-ssh-key-passphrase TUNNEL_SSH_KEY_PASSPHRASEDefault: None. Private key passphrase--tunnel-remote-port TUNNEL_REMOTE_PORTDefault: 8899. Remote port which will be forwardedlocally for proxy.--threadless          Default: False. Enabled by default on Python 3.8+(mac, linux). When disabled a new thread is spawned tohandle each client connection.--threaded            Default: True. Disabled by default on Python < 3.8 andwindows. When enabled a new thread is spawned tohandle each client connection.--num-workers NUM_WORKERSDefaults to number of CPU cores.--enable-events       Default: False. Enables core to dispatch lifecycleevents. Plugins can be used to subscribe for coreevents.--local-executor LOCAL_EXECUTORDefault: 1. Enabled by default. Use 0 to disable. Whenenabled acceptors will make use of local (sameprocess) executor instead of distributing load acrossremote (other process) executors. Enable this optionto achieve CPU affinity between acceptors andexecutors, instead of using underlying OS kernelscheduling algorithm.--backlog BACKLOG     Default: 100. Maximum number of pending connections toproxy server.--hostname HOSTNAME   Default: 127.0.0.1. Server IP address.--port PORT           Default: 8899. Server port. To listen on more ports,pass them using --ports flag.--ports PORTS [PORTS ...]Default: None. Additional ports to listen on.--port-file PORT_FILEDefault: None. Save server port numbers. Useful whenusing --port=0 ephemeral mode.--unix-socket-path UNIX_SOCKET_PATHDefault: None. Unix socket path to use. When provided--host and --port flags are ignored--num-acceptors NUM_ACCEPTORSDefaults to number of CPU cores.--version, -v         Prints proxy.py version.--log-level LOG_LEVELValid options: DEBUG, INFO (default), WARNING, ERROR,CRITICAL. Both upper and lowercase values are allowed.You may also simply use the leading character e.g.--log-level d--log-file LOG_FILE   Default: sys.stdout. Log file destination.--log-format LOG_FORMATLog format for Python logger.--open-file-limit OPEN_FILE_LIMITDefault: 1024. Maximum number of files (TCPconnections) that proxy.py can open concurrently.--plugins PLUGINS [PLUGINS ...]Comma separated plugins. You may use --plugins flagmultiple times.--enable-dashboard    Default: False. Enables proxy.py dashboard.--basic-auth BASIC_AUTHDefault: No authentication. Specify colon separateduser:password to enable basic authentication.--enable-ssh-tunnel   Default: False. Enable SSH tunnel.--work-klass WORK_KLASSDefault: proxy.http.HttpProtocolHandler. Work klass touse for work execution.--pid-file PID_FILE   Default: None. Save "parent" process ID to a file.--enable-proxy-protocolDefault: False. If used, will enable proxy protocol.Only version 1 is currently supported.--enable-conn-pool    Default: False. (WIP) Enable upstream connectionpooling.--key-file KEY_FILE   Default: None. Server key file to enable end-to-endTLS encryption with clients. If used, must also pass--cert-file.--cert-file CERT_FILEDefault: None. Server certificate to enable end-to-endTLS encryption with clients. If used, must also pass--key-file.--client-recvbuf-size CLIENT_RECVBUF_SIZEDefault: 128 KB. Maximum amount of data received fromthe client in a single recv() operation.--server-recvbuf-size SERVER_RECVBUF_SIZEDefault: 128 KB. Maximum amount of data received fromthe server in a single recv() operation.--max-sendbuf-size MAX_SENDBUF_SIZEDefault: 64 KB. Maximum amount of data to flush in asingle send() operation.--timeout TIMEOUT     Default: 10.0. Number of seconds after which aninactive connection must be dropped. Inactivity isdefined by no data sent or received by the client.--disable-http-proxy  Default: False. Whether to disableproxy.HttpProxyPlugin.--disable-headers DISABLE_HEADERSDefault: None. Comma separated list of headers toremove before dispatching client request to upstreamserver.--ca-key-file CA_KEY_FILEDefault: None. CA key to use for signing dynamicallygenerated HTTPS certificates. If used, must also pass--ca-cert-file and --ca-signing-key-file--ca-cert-dir CA_CERT_DIRDefault: ~/.proxy/certificates. Directory to storedynamically generated certificates. Also see --ca-key-file, --ca-cert-file and --ca-signing-key-file--ca-cert-file CA_CERT_FILEDefault: None. Signing certificate to use for signingdynamically generated HTTPS certificates. If used,must also pass --ca-key-file and --ca-signing-key-file--ca-file CA_FILE     Default: /Users/abhinavsingh/Dev/proxy.py/venv373/lib/python3.7/site-packages/certifi/cacert.pem. Providepath to custom CA bundle for peer certificateverification--ca-signing-key-file CA_SIGNING_KEY_FILEDefault: None. CA signing key to use for dynamicgeneration of HTTPS certificates. If used, must alsopass --ca-key-file and --ca-cert-file--auth-plugin AUTH_PLUGINDefault: proxy.http.proxy.auth.AuthPlugin. Auth pluginto use instead of default basic auth plugin.--cache-requests      Default: False. Whether to also write request packetsin the cache file.--cache-by-content-typeDefault: False. Whether to extract content by typefrom responses. Extracted content type is written tothe cache directory e.g. video.mp4.--cache-dir CACHE_DIRDefault: /Users/abhinavsingh/.proxy/cache. Flag onlyapplicable when cache plugin is used with on-diskstorage.--proxy-pool PROXY_POOLList of upstream proxies to use in the pool--enable-web-server   Default: False. Whether to enableproxy.HttpWebServerPlugin.--enable-static-serverDefault: False. Enable inbuilt static file server.Optionally, also use --static-server-dir to servestatic content from custom directory. By default,static file server serves out of installed proxy.pypython module folder.--static-server-dir STATIC_SERVER_DIRDefault: "public" folder in directory where proxy.pyis placed. This option is only applicable when staticserver is also enabled. See --enable-static-server.--min-compression-length MIN_COMPRESSION_LENGTHDefault: 20 bytes. Sets the minimum length of aresponse that will be compressed (gzipped).--enable-reverse-proxyDefault: False. Whether to enable reverse proxy core.--pac-file PAC_FILE   A file (Proxy Auto Configuration) or string to servewhen the server receives a direct file request. Usingthis option enables proxy.HttpWebServerPlugin.--pac-file-url-path PAC_FILE_URL_PATHDefault: /. Web server path to serve the PAC file.--cloudflare-dns-mode CLOUDFLARE_DNS_MODEDefault: security. Either "security" (for malwareprotection) or "family" (for malware and adult contentprotection)--filtered-upstream-hosts FILTERED_UPSTREAM_HOSTSDefault: Blocks Facebook. Comma separated list of IPv4and IPv6 addresses.--filtered-client-ips-mode FILTERED_CLIENT_IPS_MODEDefault: blacklist. Can be either "whitelist"(restrict access to specific IPs)or "blacklist" (alloweverything except specific IPs).--filtered-client-ips FILTERED_CLIENT_IPSDefault: 127.0.0.1,::1. Comma separated list of IPv4and IPv6 addresses.--filtered-url-regex-config FILTERED_URL_REGEX_CONFIGDefault: No config. Comma separated list of IPv4 andIPv6 addresses.Proxy.py not working? Report at:https://github.com/abhinavsingh/proxy.py/issues/new