Repository: mitmproxy/mitmproxy

Issue:
Requesting complete URL over https proxy results in "Invalid request scheme: https"
##### Steps to reproduce the problem:
1. Set up an HTTPS mitmproxy
2. Run the following (minimized) code against it:

``` python
import httplib
conn = httplib.HTTPSConnection('<your proxy IP>',<your proxy port>)
conn.set_tunnel('www.google.com',443)
conn.request('GET','https://www.google.com:443/')
resp = conn.getresponse()
print resp.read()
```
##### What is the expected behavior?

``` html
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
```
##### What went wrong?

``` html
<html>
            <head>
                <title>400 Bad Request</title>
            </head>
            <body>HttpException('Invalid request scheme: https',)</body>
        </html>
```
##### Any other comments?

I believe it's caused by a combination of https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/protocol/http.py#L544-L545 and https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/protocol/http.py#L502-L503. Indeed, if you modify my `GET` request above to ask for just `/` instead of `https://www.google.com:443/`, everything works fine.

I'm not sure if it's sensible to support absolute-form requests over https tunnels, but I'm getting this behavior in real code (boto for python AWS) so I don't think it's very a contrived situation.

---

mitmproxy version: 0.14.0



Discussion:
I attempted a naive patch to remove the check for `http` and the check right below it for expected form but that just led to a 400 error elsewhere.

Oh, never mind. Adjusting the two checks to allow for my situation makes the whole thing work beautifully. Not sure if it messes with other mitmproxy internal invariants though.

It actually seems to mess with the way the request goes through to the final destination though, and breaks AWS's request signing after it makes its way through the proxy.

Thanks for investigating this!

> I'm not sure if it's sensible to support absolute-form requests over https tunnels

Unfortunately, it's part of the spec that every HTTP1 request can be in absolute-form. I haven't seen it in practice before this though.

For us, it's a nontrivial UI/UX problem. Cosider an initial CONNECT request

``` http
CONNECT 10.0.0.1 HTTP/1.1
```

followed by this HTTP request:

``` http
GET http://example.com/ HTTP/1.1
Content-Length: 42
[...]
```

We now have two different destinations - which one should we show to the user? There's no guarantee that the host in the second request isn't fake and just showing "example.org" while we actually transmit data to $evilip is bad. However, we shouldn't omit the destination from the URL in the UI either.
Our current workaround is to require relative-form requests if we already have a known destination. We also catch a lot of user errors this way (e.g. people who run mitmproxy in transparent mode while configuring it on the client as well). There's also our "smart" `request.host` attribute for scripts which would need to deal with this issue as well.

TL;DR: Yes, we _should_ support this as it's perfectly valid HTTP. On the other hand, we'd need to make quite a few tradeoffs for a feature that is rarely used :confused:

> It actually seems to mess with the way the request goes through to the final destination though, and breaks AWS's request signing after it makes its way through the proxy.

I _think_ this is caused by [libmproxy/protocol/http.py#L497-L503](https://github.com/mitmproxy/mitmproxy/blob/b5d5e56fdb32717ab8f4c99c4f31c56df46afa91/libmproxy/protocol/http.py#L497-L503), which implicitly assumes relative-form requests.

One way to approach this problem would be to make the HTTP protocol support absolute-form transparent requests and add a `--relax-http` switch to mitmproxy which disables the validation part. Any better ideas?

> We now have two different destinations - which one should we show to the user?

Isn't that already a problem with standard HTTP? I can already connect to one of the IPs backing google.com and send it a request for a page on microsoft.com. The `Host` header also adds a third potential axis of variability there. Does the proxy layer really add much more?

> Any better ideas?

That seems fine, although I still don't quite see what part of the code is breaking my request signature, even if we removed the sanity checks.

Oh, I see from your link:

```
# Setting request.host also updates the host header, which we want to preserve
```

> Isn't that already a problem with standard HTTP?

It is - it also affects mitmproxy's transparent mode the same way. Our issue is that we can't distinguish if the client is now talking to a proxy (which looks the same way) or just sends an absolute-form request for no reason. Even worse, an attacker may use this to actively misguide an analyst.
For a mitmproxy user, it should always be clear where requests are going to, so we decided to just fail in these ambigious situations.

For the host header, it's a bit different in the sense as the presence of the Host header cannot be mistaken for a proxy request. Thus, you can pass `--host` to mitmproxy and we will use that as display source in transparent mode.

> That seems fine, although I still don't quite see what part of the code is breaking my request signature, even if we removed the sanity checks.

We fill in (in the absolute-form case: overwrite) attributes of the request from the connection metadata in transparent mode: [libmproxy/protocol/http.py#L497-L503](https://github.com/mitmproxy/mitmproxy/blob/b5d5e56fdb32717ab8f4c99c4f31c56df46afa91/libmproxy/protocol/http.py#L497-L503)

> We fill in (in the absolute-form case: overwrite) attributes of the request from the connection metadata in transparent mode: libmproxy/protocol/http.py#L497-L503

Yeah, sorry, I meant I couldn't tell within that what specifically was breaking boto's AWS signature. I think it's not actually any of the headers, but the path in my request that might be slightly different.

In the code I linked, `host`, `port` and `scheme` get modified - these three are attributes of the request line. In particular, `scheme` should very likely be just `http` not `https` in your case.

Changing the scheme [here](https://github.com/mitmproxy/mitmproxy/blob/b5d5e56fdb32717ab8f4c99c4f31c56df46afa91/libmproxy/protocol/http.py#L503) to `http` now just results in 502 Bad Gateway responses to all my client requests.

For debugging, I'd recommend adding breakpoints / `print(http1.assemble_request(request))` statements in `Http1Layer.read_request`/`Http1Layer.send_request`. I can't look into this myself right now, sorry.

No problem, I don't have much time right now either but I'll try to take a look a  bit later. Thanks for all your help!

@copumpkin Is this still an issue with mitmproxy 0.16?

I seem to be hitting the same issue trying to use mitmproxy with Google Drive Sync Client:

This is the error messages in the Google Drive Sync Client logs:

```
2016-07-05 11:34:46,880 +1000 WARNING pid=62177 140735142223872:MainThread      proxy_manager.pyo:420 using env setting for proxy: 'https://XXX.XXX.XXX.XXX:8080'
2016-07-05 11:34:46,881 +1000 INFO pid=62177 140735142223872:MainThread      service.pyo:132 Creating singleton service instance of '<class 'common.net.dapper.DapperService'>'
2016-07-05 11:34:46,881 +1000 INFO pid=62177 140735142223872:MainThread      sync_http_client.pyo:235 Tunneling connection to accounts.google.com:443 through 104.196.40.101:8080.
2016-07-05 11:34:48,660 +1000 ERROR pid=62177 140735142223872:MainThread      oauth.pyo:138 OAuth failure. Error on request. Response 400. <html>
            <head>
                <title>400 Bad Request</title>
            </head>
            <body>HttpException('Invalid request scheme: https',)</body>
        </html>
2016-07-05 11:34:48,661 +1000 ERROR pid=62177 140735142223872:MainThread      fatal_exceptions.pyo:55 Ignoring non-fatal error: Unauthorized: OAuth failure. Error on request. Response 400. <html>
            <head>
                <title>400 Bad Request</title>
            </head>
            <body>HttpException('Invalid request scheme: https',)</body>
        </html>, Traceback:
  File "common/auth/oauth.pyo", line 136, in GetAuthToken
  File "common/auth/oauth.pyo", line 196, in _AuthorizeOAuth2Token
  File "common/auth/oauth2_token.pyo", line 143, in AcquireAccessToken
  File "common/auth/oauth2_token.pyo", line 199, in _SendRequest

2016-07-05 11:37:19,349 +1000 INFO pid=62177 140735142223872:MainThread      service.pyo:132 Creating singleton service instance of '<class 'controls.selective_sync_stats.SelectiveSyncStatsTracker'>'
2016-07-05 11:37:19,395 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  pause_manager.pyo:106 Adding pause reason USER
2016-07-05 11:37:19,419 +1000 INFO pid=62177 123145330839552:RunAsync-_PerformAsyncSignOutTasks-7 service.pyo:132 Creating singleton service instance of '<class 'common.telemetry.impression_logger_manager.ImpressionLoggerManager'>'
2016-07-05 11:37:19,428 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  sync_app.pyo:2057 Sign out remove token success: True
2016-07-05 11:37:19,432 +1000 ERROR pid=62177 123145330839552:RunAsync-_PerformAsyncSignOutTasks-7 fatal_exceptions.pyo:55 Ignoring non-fatal error: ImpressionLoggerManagerStateError: Uploader not configured., Traceback:
  File "common/exception_utils.pyo", line 72, in SuppressExceptions
  File "common/threads/run_async.pyo", line 55, in ErrorHandler
  File "common/sync_app.pyo", line 1978, in _PerformAsyncSignOutTasks
  File "common/telemetry/impression_logger_manager.pyo", line 284, in UploadLogsIfEnabled

2016-07-05 11:37:19,434 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  sync_app.pyo:1992 Stopping threads
2016-07-05 11:37:19,434 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  thread_manager.pyo:116 Stop was called
2016-07-05 11:37:19,434 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  thread_manager.pyo:253 Asking <RunFunctionThread(_LaunchSyncApp, started 123145326632960)> with tid 123145326632960 to stop.
2016-07-05 11:37:19,435 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  thread_manager.pyo:264 Stop completed successfully.
2016-07-05 11:37:19,437 +1000 INFO pid=62177 123145326632960:_LaunchSyncApp  utils.pyo:226 set called on <common.utils.LoggingEvent object at 0x10fb9cd10> from:
  File "threading.pyo", line 783, in __bootstrap
   File "threading.pyo", line 810, in __bootstrap_inner
   File "common/threads/threads.pyo", line 93, in run
   File "threading.pyo", line 763, in run
   File "common/threads/threads.pyo", line 83, in AutoreleasePoolWrappedTarget
   File "common/sync_client_thread.pyo", line 50, in RunSyncClientThread
   File "common/sync_client_thread.pyo", line 326, in Run
   File "osx/mac_sync_app_controller.pyo", line 178, in _LaunchSyncApp
   File "common/sync_app.pyo", line 664, in LaunchThreads
   File "common/persistence/sqlite.pyo", line 127, in Wrapper
   File "common/sync_app.pyo", line 689, in _LaunchThreads
   File "common/sync_app.pyo", line 846, in _RunSetupAndSignIn
   File "common/sync_app.pyo", line 2061, in SignOut
   File "common/sync_app.pyo", line 1993, in _SignOutThreadManagerAndDropDatabases
   File "common/thread_manager.pyo", line 136, in StopThreadsAndWait
   File "common/thread_manager.pyo", line 124, in Stop
   File "common/thread_manager.pyo", line 391, in Stop
   File "common/utils.pyo", line 226, in set
```

@Kriechi Also, to clarify - yes, I believe this is still an issue in mitmproxy 0.16 - I tested against `HEAD`, and I hit this issue =(.

(The latest version on Pypi is mitmproxy 0.17, right?)

Just bumping - is this something you guys are working on =).

Any idea of timeframe? (Cool if there's not, just trying to set my expectations).

There have been some major changes regarding first-line format in mitmproxy.
Is this still an issue, or has v1.0 solved it?

To be honest, going through the thread here again:
It seems we are dealing with a very uncommon use case. Only very recently did e.g. curl start to support HTTPS proxies. Or am I missing some context from the original problem here?
This is still an issue I believe, and it's not related to HTTP**S** proxies. :-)
If a client CONNECTs to a remote server and then sends an absolute-form request (e.g. `CONNECT http://example.com:80/`, then `GET http://example.com/foo`), we currently fail deliberately. This is because the host in the CONNECT request and the GET request do not necessarily have to be the same and we just have one `.host` attribute. Generally speaking, an unexpected first-line-format is most of the times a sign of misconfiguration (the RFC permits absolute form for all requests though).
There are two things I'd love to see:

1. First, always accept requests in absolute-form going to the same host. For example, after `CONNECT example.com`, we should accept `GET http://example.com/foo`, but not `GET http://example.org/foo`. This would for example fix this issue.
2. Second, as you suggested, add a `--relax-http-form-validation` switch that disables the form enforcement.
any update on this issue?
Same issue when requesting in emacs here, using mitmproxy 4.0.1.
`--relax-http-form-validation` whats the new version of this? This param doesnot work in the newest binary
I would love to see a fix for this issue as well. The nodeJS axios library is fairly well known, and it allows requesting complete URLs. There are calls for axios buried under other utilities that implement sigV4 for AWS that it would be very nice to be able to consume and use through mitmproxy.
Just hit this with Emacs 26.1 (url.el), mitmproxy 4.0.4, trying to debug magit/forge#55 with mitmproxy.

Resolving patch:
diff --git a/mitmproxy/options.py b/mitmproxy/options.py
--- a/mitmproxy/options.py
+++ b/mitmproxy/options.py
@@ -181,5 +181,11 @@ def __init__(self, **kwargs) -> None:
             TLS key size for certificates and CA.
             """
         )
+        self.add_option(
+            "relax_http_form_validation", bool, False,
+            """
+            Disable HTTP form validation.
+            """
+        )
 
         self.update(**kwargs)
diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py
--- a/mitmproxy/proxy/protocol/http.py
+++ b/mitmproxy/proxy/protocol/http.py
@@ -267,10 +267,12 @@ def _process_flow(self, f):
                     self.send_error_response(400, msg)
                     return False
 
-            validate_request_form(self.mode, request)
+            if not self.config.options.relax_http_form_validation:
+                validate_request_form(self.mode, request)
             self.channel.ask("requestheaders", f)
             # Re-validate request form in case the user has changed something.
-            validate_request_form(self.mode, request)
+            if not self.config.options.relax_http_form_validation:
+                validate_request_form(self.mode, request)
 
             if request.headers.get("expect", "").lower() == "100-continue":
                 # TODO: We may have to use send_response_headers for HTTP2


Test patch: