Repository: mitmproxy/mitmproxy

Issue:
har_extractor errors
Hi,

Firstly, I like to say your work on mitmproxy is awesome!  I have a question with the har_extractor for it though.

I'm running a java test application does that has to various things to get specific events to be triggered on multiple platforms (web, native apps, etc)

I then need to capture these network calls and verify the contents. As these calls are https, I'm trying to use mitmdump to dump logs in har format. So my java app can start mitmdump and then read the logs to verify click and tracking events sent in the background.

I was very excited when I saw the har extractor but when I use it, i keep getting errors.

1.) am i using it correctly.
I'm currently doing it this way.

mitmdump -w out/out.dmp -s "out/har_extractor.py --out/har.har"

Then when I load the page i get a variety of errors. and nothing written to my requested output har.har file.

Can you please help me out?

Best,
Nelson

Here are the two errors I"m getting 

```
127.0.0.1 GET https://pagead2.googlesyndication.com/pagead/gen_204?id=ah&v=349&avi=BTJ8BlT74VZurCYuYkgKdzZSYBgAAAAAQATgByAEC4AQCiAWjzJoEoAY-&adk=0&pmd=903&pd=903&fd=1
 << 204 No Content 0B
Script error:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/libmproxy/script.py", line 123, in run
    return (True, f(self.ctx, *args, **kwargs))
  File "out/har_extractor.py", line 213, in response
    elif context.HARLog.get_page_ref(flow.request.headers.get('Referer')) is not None:
  File "out/har_extractor.py", line 52, in get_page_ref
    return self.__page_ref__.get(page, None)
TypeError: unhashable type: 'list'
```

and 

```
127.0.0.1 GET https://pagead2.googlesyndication.com/pagead/gen_204?id=sodar&v=5&t=2&bgai=B_TMolT74VZurCYuYkgKdzZSYBgAAAAA4AeAEAg&bg=!fH9EMCiIOIGddMgCAAAAHVIAAAABKgDrLq8aJ3HDu35G_dL4p9Rd_0UjdyteH5e--Qm2lvQzYas8iMaXH_64LJ-ldQUWyAtqb7L87kMJ7F8OwNDGuxfJ5LoHniT4o1d4YhCeubKVyQghXgsPZkx3ITCtIqNHc28623JmVTuII6AEjs7nb5nn5msqQPVpF_b4x5x3dFf6wzD8U7bWylv6ukc9mEpT3T5l7t3VtPjJK8mKsfjufqEUln5XTVBIjh7RCcecuGUGTNzFMzTOB1299zxo4XRbhKiqMIyP8XTAPt8h72CAV4rRtEkBj29Ju045wrvc6vvQ_I9ZBHgDPFrJa6vSjQ
 << 204 No Content 0B
Script error:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/libmproxy/script.py", line 123, in run
    return (True, f(self.ctx, *args, **kwargs))
  File "out/har_extractor.py", line 143, in response
    for k, v in (flow.request.get_cookies() or {}).iteritems()]
AttributeError: 'ODict' object has no attribute 'iteritems'
```



Discussion:
The first issue you're seeing is caused by the fact that you're using the example from master while you have 0.13 installed.

The second error is _probably_ an issue in the script itself, you _should_ be good by just replacing `.iteritems()` with `.items()`. The har extractor has been contributed by @JustusW - maybe he can chip in?

Cheers,
Max

@mhils this error looks as if request.get_cookies suddenly returns an ODict that does not support the full MutableMapping interface anymore?

To the best of my knowledge, ODict never supported the full MutableMapping interface (and also has some slightly different behaviour, i.e. `del` does not throw an error if the key doesn't exist).

Well, it needs key, value pairs, so items _should_ be fine, it is a bit irritating since either this hasn't worked for quite some time (which would be strange seeing I use this in a production environment on a more or less hourly basis) or there has been a very recent change to either the ODict returned or the return value changed into an odict.

Please mind that this would have to be changed in at least two places where cookies are extracted.


Resolving patch:
diff --git a/examples/har_extractor.py b/examples/har_extractor.py
--- a/examples/har_extractor.py
+++ b/examples/har_extractor.py
@@ -127,11 +127,11 @@ def response(context, flow):
         tz=utc).isoformat()
 
     request_query_string = [{"name": k, "value": v}
-                            for k, v in flow.request.get_query()]
+                            for k, v in flow.request.query]
     request_http_version = flow.request.http_version
     # Cookies are shaped as tuples by MITMProxy.
     request_cookies = [{"name": k.strip(), "value": v[0]}
-                       for k, v in (flow.request.get_cookies() or {}).iteritems()]
+                       for k, v in flow.request.cookies.items()]
     request_headers = [{"name": k, "value": v} for k, v in flow.request.headers]
     request_headers_size = len(str(flow.request.headers))
     request_body_size = len(flow.request.content)
@@ -139,7 +139,7 @@ def response(context, flow):
     response_http_version = flow.response.http_version
     # Cookies are shaped as tuples by MITMProxy.
     response_cookies = [{"name": k.strip(), "value": v[0]}
-                        for k, v in (flow.response.get_cookies() or {}).iteritems()]
+                        for k, v in flow.response.cookies.items()]
     response_headers = [{"name": k, "value": v}
                         for k, v in flow.response.headers]
     response_headers_size = len(str(flow.response.headers))


Test patch: