Repository: mitmproxy/mitmproxy

Issue:
Implement `share` command
@MatthewShao implemented a really cool static viewer based on mitmweb last summer: http://honeynet.org/node/1359.

Long story short, one can upload a mitmproxy flow file at share.mitmproxy.org and will get a shareable link in return. 😃 To make this more accessible, I think it would be awesome to have a mitmproxy command that takes a bunch of flows as argument, uploads them directly from within mitmproxy, and returns a URL to share.
Implement `share` command
@MatthewShao implemented a really cool static viewer based on mitmweb last summer: http://honeynet.org/node/1359.

Long story short, one can upload a mitmproxy flow file at share.mitmproxy.org and will get a shareable link in return. 😃 To make this more accessible, I think it would be awesome to have a mitmproxy command that takes a bunch of flows as argument, uploads them directly from within mitmproxy, and returns a URL to share.


Discussion:
For some related code, see e.g. the save command: https://github.com/mitmproxy/mitmproxy/blob/67300fab3119f055a6cbb43b7cbc9d33e849d165/mitmproxy/addons/save.py#L53-L67 
It would be nice to use HTTP upload functionality from Python's stdlib opposed to requests to keep our library footprint small.
@mhils Just want to clarify. I press, for example, `u` button, mitmproxy takes all available flows, uploads them at _share.mitmproxy.org_ and then gets _url_, displaying it in Status Bar.  But how will we be able to copy it and use somewhere?
I've been thinking about this as well. It's a very exciting direction. This is certainly a 4.0 or beyond feature - we have quite a bit of work/thinking to do about the server component here.
@kajojify: For now, let's just add a command that can be manually invoked (press `:` and then enter the command and a flowspec) and logs the URL.
Hi, I would like to work on it :) Will keep you updated as I progress
I am almost done, one doubt I have is that since the data is being encoded by multipart/form-data , I need to load the file into the memory first, can I use 'poster' to achieve this or go forward with standard libraries?
@mhils 

![mitm-upload half](https://user-images.githubusercontent.com/19341613/35057494-20107d5a-fbdb-11e7-8295-b7a2216dc9df.png)

Hi,

stdlib please. As in your screenshot, we should accept a FlowSpec, i.e. the command will be given a list of flows, which it can then serialize in memory to a BytesIO instance (see [mitmproxy.io](https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/io/io.py)), which we then can upload. :)

We want filesizes to be reasonable anyway, so in-memory is not an issue here.
Hi,
So the approach I am using is that I am saving the flows in a temporary file, (the name will be the unique 7 letter id), the doubt I have is that to send a POST request I need to read the contents of the file so should I use `read_flows_from_paths(paths)` or
```
path = os.path.expanduser(path)
with open(path, "rb") as f:
        content  = f.read()
```
and use `content` to send the POST request?
For some related code, see e.g. the save command: https://github.com/mitmproxy/mitmproxy/blob/67300fab3119f055a6cbb43b7cbc9d33e849d165/mitmproxy/addons/save.py#L53-L67 
It would be nice to use HTTP upload functionality from Python's stdlib opposed to requests to keep our library footprint small.
@mhils Just want to clarify. I press, for example, `u` button, mitmproxy takes all available flows, uploads them at _share.mitmproxy.org_ and then gets _url_, displaying it in Status Bar.  But how will we be able to copy it and use somewhere?
I've been thinking about this as well. It's a very exciting direction. This is certainly a 4.0 or beyond feature - we have quite a bit of work/thinking to do about the server component here.
@kajojify: For now, let's just add a command that can be manually invoked (press `:` and then enter the command and a flowspec) and logs the URL.
Hi, I would like to work on it :) Will keep you updated as I progress
I am almost done, one doubt I have is that since the data is being encoded by multipart/form-data , I need to load the file into the memory first, can I use 'poster' to achieve this or go forward with standard libraries?
@mhils 

![mitm-upload half](https://user-images.githubusercontent.com/19341613/35057494-20107d5a-fbdb-11e7-8295-b7a2216dc9df.png)

Hi,

stdlib please. As in your screenshot, we should accept a FlowSpec, i.e. the command will be given a list of flows, which it can then serialize in memory to a BytesIO instance (see [mitmproxy.io](https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/io/io.py)), which we then can upload. :)

We want filesizes to be reasonable anyway, so in-memory is not an issue here.
Hi,
So the approach I am using is that I am saving the flows in a temporary file, (the name will be the unique 7 letter id), the doubt I have is that to send a POST request I need to read the contents of the file so should I use `read_flows_from_paths(paths)` or
```
path = os.path.expanduser(path)
with open(path, "rb") as f:
        content  = f.read()
```
and use `content` to send the POST request?

Resolving patch:
diff --git a/mitmproxy/addons/__init__.py b/mitmproxy/addons/__init__.py
--- a/mitmproxy/addons/__init__.py
+++ b/mitmproxy/addons/__init__.py
@@ -20,6 +20,7 @@
 from mitmproxy.addons import streambodies
 from mitmproxy.addons import save
 from mitmproxy.addons import upstream_auth
+from mitmproxy.addons import share
 
 
 def default_addons():
@@ -46,4 +47,5 @@ def default_addons():
         streambodies.StreamBodies(),
         save.Save(),
         upstream_auth.UpstreamAuth(),
+        share.Share()
     ]
diff --git a/mitmproxy/addons/save.py b/mitmproxy/addons/save.py
--- a/mitmproxy/addons/save.py
+++ b/mitmproxy/addons/save.py
@@ -61,8 +61,8 @@ def save(self, flows: typing.Sequence[flow.Flow], path: mitmproxy.types.Path) ->
         except IOError as v:
             raise exceptions.CommandError(v) from v
         stream = io.FlowWriter(f)
-        for i in flows:
-            stream.add(i)
+        for x in flows:
+            stream.add(x)
         f.close()
         ctx.log.alert("Saved %s flows." % len(flows))
 
diff --git a/mitmproxy/addons/share.py b/mitmproxy/addons/share.py
new file mode 100644
--- /dev/null
+++ b/mitmproxy/addons/share.py
@@ -0,0 +1,79 @@
+import typing
+import random
+import string
+import io
+import http.client
+
+from mitmproxy import command
+import mitmproxy.io
+from mitmproxy import ctx
+from mitmproxy import flow
+from mitmproxy.net.http import status_codes
+
+
+class Share:
+    def encode_multipart_formdata(self, filename: str, content: bytes) -> typing.Tuple[str, bytes]:
+        params = {"key": filename, "acl": "bucket-owner-full-control", "Content-Type": "application/octet-stream"}
+        LIMIT = b'---------------------------198495659117975628761412556003'
+        CRLF = b'\r\n'
+        l = []
+        for (key, value) in params.items():
+            l.append(b'--' + LIMIT)
+            l.append(b'Content-Disposition: form-data; name="%b"' % key.encode("utf-8"))
+            l.append(b'')
+            l.append(value.encode("utf-8"))
+        l.append(b'--' + LIMIT)
+        l.append(b'Content-Disposition: form-data; name="file"; filename="%b"' % filename.encode("utf-8"))
+        l.append(b'Content-Type: application/octet-stream')
+        l.append(b'')
+        l.append(content)
+        l.append(b'--' + LIMIT + b'--')
+        l.append(b'')
+        body = CRLF.join(l)
+        content_type = 'multipart/form-data; boundary=%s' % LIMIT.decode("utf-8")
+        return content_type, body
+
+    def post_multipart(self, host: str, filename: str, content: bytes) -> str:
+        """
+        Upload flows to the specified S3 server.
+
+        Returns:
+            - The share URL, if upload is successful.
+        Raises:
+            - IOError, otherwise.
+        """
+        content_type, body = self.encode_multipart_formdata(filename, content)
+        conn = http.client.HTTPConnection(host)  # FIXME: This ultimately needs to be HTTPSConnection
+        headers = {'content-type': content_type}
+        try:
+            conn.request("POST", "", body, headers)
+            resp = conn.getresponse()
+        except Exception as v:
+            raise IOError(v)
+        finally:
+            conn.close()
+        if resp.status != 204:
+            if resp.reason:
+                reason = resp.reason
+            else:
+                reason = status_codes.RESPONSES.get(resp.status, str(resp.status))
+            raise IOError(reason)
+        return "https://share.mitmproxy.org/%s" % filename
+
+    @command.command("share.flows")
+    def share(self, flows: typing.Sequence[flow.Flow]) -> None:
+        u_id = "".join(random.choice(string.ascii_lowercase + string.digits)for _ in range(7))
+        f = io.BytesIO()
+        stream = mitmproxy.io.FlowWriter(f)
+        for x in flows:
+            stream.add(x)
+        f.seek(0)
+        content = f.read()
+        try:
+            res = self.post_multipart('upload.share.mitmproxy.org.s3.amazonaws.com', u_id, content)
+        except IOError as v:
+            ctx.log.warn("%s" % v)
+        else:
+            ctx.log.alert("%s" % res)
+        finally:
+            f.close()
\ No newline at end of file


Test patch:
diff --git a/test/mitmproxy/addons/test_share.py b/test/mitmproxy/addons/test_share.py
new file mode 100644
--- /dev/null
+++ b/test/mitmproxy/addons/test_share.py
@@ -0,0 +1,34 @@
+from unittest import mock
+import http.client
+
+from mitmproxy.test import taddons
+from mitmproxy.test import tflow
+
+from mitmproxy.addons import share
+from mitmproxy.addons import view
+
+
+def test_share_command():
+    with mock.patch('mitmproxy.addons.share.http.client.HTTPConnection') as mock_http:
+        sh = share.Share()
+        with taddons.context() as tctx:
+            mock_http.return_value.getresponse.return_value = mock.MagicMock(status=204, reason="No Content")
+            sh.share([tflow.tflow(resp=True)])
+            assert tctx.master.has_log("https://share.mitmproxy.org/")
+
+            mock_http.return_value.getresponse.return_value = mock.MagicMock(status=403, reason="Forbidden")
+            sh.share([tflow.tflow(resp=True)])
+            assert tctx.master.has_log("Forbidden")
+
+            mock_http.return_value.getresponse.return_value = mock.MagicMock(status=404, reason="")
+            sh.share([tflow.tflow(resp=True)])
+            assert tctx.master.has_log("Not Found")
+
+            mock_http.return_value.request.side_effect = http.client.CannotSendRequest("Error in sending req")
+            sh.share([tflow.tflow(resp=True)])
+            assert tctx.master.has_log("Error in sending req")
+
+            v = view.View()
+            tctx.master.addons.add(v)
+            tctx.master.addons.add(sh)
+            tctx.master.commands.call_args("share.flows", ["@shown"])