Repository: mitmproxy/mitmproxy

Issue:
HEAD method not allowed to access mitm.it
##### Steps to reproduce the problem:

1. Using mitmdump in transparent mode, with on-boarding port on 6969 (did not test on 80)
/mitmdump -T --host --insecure --onboarding-port 6969

2. On iPad + Chrome + Transparent mode, access to http://mitm.it:6969

3. An HEAD /cert/pem is sent, receive an "405 Method Not Allowed"

Any idea of any workaround?


##### Any other comments? What have you tried so far?

Exact packet:
Hypertext Transfer Protocol
    HEAD /cert/pem HTTP/1.1\r\n
    Host: mitm.it:6969\r\n
    Connection: keep-alive\r\n
    User-Agent: Mozilla/5.0 (iPad; CPU OS 9_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/55.0.2883.79 Mobile/13C75 Safari/601.1.46\r\n
    Accept-Encoding: gzip, deflate, sdch\r\n
    Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4\r\n
    \r\n



Answer:
Hypertext Transfer Protocol
    HTTP/1.1 405 Method Not Allowed\r\n
    Server: TornadoServer/4.4.2\r\n
    Content-Length: 87\r\n
    Date: Tue, 09 May 2017 13:35:38 GMT\r\n
    Content-Type: text/html; charset=UTF-8\r\n
    \r\n



##### System information

mitmdump --version
Mitmproxy version: 2.0.1 (release version) Precompiled Binary
Python version: 3.5.2
Platform: Darwin-16.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.12.4 ('', '', '') x86_64

HEAD method not allowed to access mitm.it
##### Steps to reproduce the problem:

1. Using mitmdump in transparent mode, with on-boarding port on 6969 (did not test on 80)
/mitmdump -T --host --insecure --onboarding-port 6969

2. On iPad + Chrome + Transparent mode, access to http://mitm.it:6969

3. An HEAD /cert/pem is sent, receive an "405 Method Not Allowed"

Any idea of any workaround?


##### Any other comments? What have you tried so far?

Exact packet:
Hypertext Transfer Protocol
    HEAD /cert/pem HTTP/1.1\r\n
    Host: mitm.it:6969\r\n
    Connection: keep-alive\r\n
    User-Agent: Mozilla/5.0 (iPad; CPU OS 9_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/55.0.2883.79 Mobile/13C75 Safari/601.1.46\r\n
    Accept-Encoding: gzip, deflate, sdch\r\n
    Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4\r\n
    \r\n



Answer:
Hypertext Transfer Protocol
    HTTP/1.1 405 Method Not Allowed\r\n
    Server: TornadoServer/4.4.2\r\n
    Content-Length: 87\r\n
    Date: Tue, 09 May 2017 13:35:38 GMT\r\n
    Content-Type: text/html; charset=UTF-8\r\n
    \r\n



##### System information

mitmdump --version
Mitmproxy version: 2.0.1 (release version) Precompiled Binary
Python version: 3.5.2
Platform: Darwin-16.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.12.4 ('', '', '') x86_64

HEAD method not allowed to access mitm.it
##### Steps to reproduce the problem:

1. Using mitmdump in transparent mode, with on-boarding port on 6969 (did not test on 80)
/mitmdump -T --host --insecure --onboarding-port 6969

2. On iPad + Chrome + Transparent mode, access to http://mitm.it:6969

3. An HEAD /cert/pem is sent, receive an "405 Method Not Allowed"

Any idea of any workaround?


##### Any other comments? What have you tried so far?

Exact packet:
Hypertext Transfer Protocol
    HEAD /cert/pem HTTP/1.1\r\n
    Host: mitm.it:6969\r\n
    Connection: keep-alive\r\n
    User-Agent: Mozilla/5.0 (iPad; CPU OS 9_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/55.0.2883.79 Mobile/13C75 Safari/601.1.46\r\n
    Accept-Encoding: gzip, deflate, sdch\r\n
    Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4\r\n
    \r\n



Answer:
Hypertext Transfer Protocol
    HTTP/1.1 405 Method Not Allowed\r\n
    Server: TornadoServer/4.4.2\r\n
    Content-Length: 87\r\n
    Date: Tue, 09 May 2017 13:35:38 GMT\r\n
    Content-Type: text/html; charset=UTF-8\r\n
    \r\n



##### System information

mitmdump --version
Mitmproxy version: 2.0.1 (release version) Precompiled Binary
Python version: 3.5.2
Platform: Darwin-16.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.12.4 ('', '', '') x86_64



Discussion:
AFAIK the mitm.it just implements enough to serve the page.
Any specific reason why you want to use a HEAD request?
Or is this an intrinsic behaviour of Chrome for iOS? Why are they doing it?
Actually it makes both requests in a row:
GET, I receive a 400
HEAD, I receive 405

<img width="1173" alt="screenshot" src="https://cloud.githubusercontent.com/assets/15690818/25855704/422c2d22-34d4-11e7-9392-7255eafd38cf.png">

It works well on Safari on iPad

Ok - thanks for the wireshark trace.
Looks like the regular GET still works ok.

Not sure now why you are seeing a HEAD request as well.
But I guess there is no harm in implementing it here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L41-L57
and here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L60-L76

PRs are welcome!
You mean something like `def head(self): ...` with exact same definition as def get(self)?

Or it's not simple as that?
More or less.
The important bit is probably `self.set_header("Content-Length", "7")`
AFAIK the mitm.it just implements enough to serve the page.
Any specific reason why you want to use a HEAD request?
Or is this an intrinsic behaviour of Chrome for iOS? Why are they doing it?
Actually it makes both requests in a row:
GET, I receive a 400
HEAD, I receive 405

<img width="1173" alt="screenshot" src="https://cloud.githubusercontent.com/assets/15690818/25855704/422c2d22-34d4-11e7-9392-7255eafd38cf.png">

It works well on Safari on iPad

Ok - thanks for the wireshark trace.
Looks like the regular GET still works ok.

Not sure now why you are seeing a HEAD request as well.
But I guess there is no harm in implementing it here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L41-L57
and here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L60-L76

PRs are welcome!
You mean something like `def head(self): ...` with exact same definition as def get(self)?

Or it's not simple as that?
More or less.
The important bit is probably `self.set_header("Content-Length", "7")`
AFAIK the mitm.it just implements enough to serve the page.
Any specific reason why you want to use a HEAD request?
Or is this an intrinsic behaviour of Chrome for iOS? Why are they doing it?
Actually it makes both requests in a row:
GET, I receive a 400
HEAD, I receive 405

<img width="1173" alt="screenshot" src="https://cloud.githubusercontent.com/assets/15690818/25855704/422c2d22-34d4-11e7-9392-7255eafd38cf.png">

It works well on Safari on iPad

Ok - thanks for the wireshark trace.
Looks like the regular GET still works ok.

Not sure now why you are seeing a HEAD request as well.
But I guess there is no harm in implementing it here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L41-L57
and here
https://github.com/mitmproxy/mitmproxy/blob/81d68aa564d640a64c5394c0fcd648433e100090/mitmproxy/addons/onboardingapp/app.py#L60-L76

PRs are welcome!
You mean something like `def head(self): ...` with exact same definition as def get(self)?

Or it's not simple as that?
More or less.
The important bit is probably `self.set_header("Content-Length", "7")`

Resolving patch:
diff --git a/mitmproxy/addons/onboardingapp/app.py b/mitmproxy/addons/onboardingapp/app.py
--- a/mitmproxy/addons/onboardingapp/app.py
+++ b/mitmproxy/addons/onboardingapp/app.py
@@ -44,6 +44,18 @@ class PEM(tornado.web.RequestHandler):
     def filename(self):
         return config.CONF_BASENAME + "-ca-cert.pem"
 
+    def head(self):
+        p = os.path.join(self.request.master.options.cadir, self.filename)
+        p = os.path.expanduser(p)
+        content_length = os.path.getsize(p)
+
+        self.set_header("Content-Type", "application/x-x509-ca-cert")
+        self.set_header(
+            "Content-Disposition",
+            "inline; filename={}".format(
+                self.filename))
+        self.set_header("Content-Length", content_length)
+
     def get(self):
         p = os.path.join(self.request.master.options.cadir, self.filename)
         p = os.path.expanduser(p)
@@ -63,6 +75,19 @@ class P12(tornado.web.RequestHandler):
     def filename(self):
         return config.CONF_BASENAME + "-ca-cert.p12"
 
+    def head(self):
+        p = os.path.join(self.request.master.options.cadir, self.filename)
+        p = os.path.expanduser(p)
+        content_length = os.path.getsize(p)
+
+        self.set_header("Content-Type", "application/x-pkcs12")
+        self.set_header(
+            "Content-Disposition",
+            "inline; filename={}".format(
+                self.filename))
+
+        self.set_header("Content-Length", content_length)
+
     def get(self):
         p = os.path.join(self.request.master.options.cadir, self.filename)
         p = os.path.expanduser(p)


Test patch:
diff --git a/test/mitmproxy/addons/test_onboarding.py b/test/mitmproxy/addons/test_onboarding.py
--- a/test/mitmproxy/addons/test_onboarding.py
+++ b/test/mitmproxy/addons/test_onboarding.py
@@ -1,5 +1,8 @@
+import pytest
+
 from mitmproxy.addons import onboarding
 from mitmproxy.test import taddons
+from mitmproxy import options
 from .. import tservers
 
 
@@ -12,10 +15,21 @@ def test_basic(self):
             tctx.configure(self.addons()[0])
             assert self.app("/").status_code == 200
 
-    def test_cert(self):
+    @pytest.mark.parametrize("ext", ["pem", "p12"])
+    def test_cert(self, ext):
+        with taddons.context() as tctx:
+            tctx.configure(self.addons()[0])
+            resp = self.app("/cert/%s" % ext)
+            assert resp.status_code == 200
+            assert resp.content
+
+    @pytest.mark.parametrize("ext", ["pem", "p12"])
+    def test_head(self, ext):
         with taddons.context() as tctx:
             tctx.configure(self.addons()[0])
-            for ext in ["pem", "p12"]:
-                resp = self.app("/cert/%s" % ext)
+            p = self.pathoc()
+            with p.connect():
+                resp = p.request("head:'http://%s/cert/%s'" % (options.APP_HOST, ext))
                 assert resp.status_code == 200
-                assert resp.content
+                assert "Content-Length" in resp.headers
+                assert not resp.content