Repository: mitmproxy/mitmproxy

Issue:
Dump files not standard across versions
#### Problem Description

Dump files generated by mitmdump 5.2 aren't compatible with 5.3.x or 6.x.x.

Whats worse is that they're partially compatible, somehow the newer versions read one request in the file and not the rest of it.

#### Proposal

I get that maintaining backward compatibility is super hard and don't expect the project to start doing this.
It would be nice if users were informed of intricacies like this one.

#### Alternatives

Versioning info in Dump files and an error message when newer versions try to read them would be great.

Backward compatibility would be the best option of course, but trying to debug something like this is very hard.
Hopefully too many people haven't spent time trying to figure this out.



Discussion:
Old dumpfiles can generally be read with newer versions since mitmproxy 0.11 or so. If that doesn't work for a particular file it's a bug and we need more details. :)
Sure!

[This](https://github.com/amoghbl1/mitm-dumps/blob/master/example.dump) dump works fine with mitmdump 5.2 but not with versions after it.

Resolving patch:
diff --git a/mitmproxy/io/compat.py b/mitmproxy/io/compat.py
--- a/mitmproxy/io/compat.py
+++ b/mitmproxy/io/compat.py
@@ -312,6 +312,10 @@ def convert_12_13(data):
 def convert_13_14(data):
     data["version"] = 14
     data["comment"] = ""
+    # bugfix for https://github.com/mitmproxy/mitmproxy/issues/4576
+    if data.get("response", None) and data["response"]["timestamp_start"] is None:
+        data["response"]["timestamp_start"] = data["request"]["timestamp_end"]
+        data["response"]["timestamp_end"] = data["request"]["timestamp_end"] + 1
     return data
 
 


Test patch:
diff --git a/test/helper_tools/dumperview.py b/test/helper_tools/dumperview.py
--- a/test/helper_tools/dumperview.py
+++ b/test/helper_tools/dumperview.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import click
 
 from mitmproxy.addons import dumper
diff --git a/test/helper_tools/inspect_dumpfile.py b/test/helper_tools/inspect_dumpfile.py
--- a/test/helper_tools/inspect_dumpfile.py
+++ b/test/helper_tools/inspect_dumpfile.py
@@ -1,8 +1,9 @@
+#!/usr/bin/env python3
 from pprint import pprint
 
 import click
 
-from mitmproxy import tnetstring
+from mitmproxy.io import tnetstring
 
 
 def read_tnetstring(input):
diff --git a/test/mitmproxy/data/dumpfile-018.mitm b/test/mitmproxy/data/dumpfile-018.mitm
--- a/test/mitmproxy/data/dumpfile-018.mitm
+++ b/test/mitmproxy/data/dumpfile-018.mitm
@@ -1,54 +1,4 @@
-7816:4:type;4:http;2:id;36:55367415-10f5-4938-b69f-8a523394f947;8:response;1851:15:timestamp_start;17:1482157524.361187^12:http_version;8:HTTP/1.1,7:content;1270:<!doctype html>
-<html>
-<head>
-    <title>Example Domain</title>
-
-    <meta charset="utf-8" />
-    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
-    <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <style type="text/css">
-    body {
-        background-color: #f0f0f2;
-        margin: 0;
-        padding: 0;
-        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
-        
-    }
-    div {
-        width: 600px;
-        margin: 5em auto;
-        padding: 50px;
-        background-color: #fff;
-        border-radius: 1em;
-    }
-    a:link, a:visited {
-        color: #38488f;
-        text-decoration: none;
-    }
-    @media (max-width: 700px) {
-        body {
-            background-color: #fff;
-        }
-        div {
-            width: auto;
-            margin: 0 auto;
-            border-radius: 0;
-            padding: 1em;
-        }
-    }
-    </style>    
-</head>
-
-<body>
-<div>
-    <h1>Example Domain</h1>
-    <p>This domain is established to be used for illustrative examples in documents. You may use this
-    domain in examples without prior coordination or asking for permission.</p>
-    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
-</div>
-</body>
-</html>
-,13:timestamp_end;17:1482157524.361187^11:status_code;3:200#6:reason;2:OK,7:headers;410:25:13:Accept-Ranges,5:bytes,]35:13:Cache-Control,14:max-age=604800,]28:12:Content-Type,9:text/html,]40:4:Date,29:Mon, 19 Dec 2016 14:25:24 GMT,]22:4:Etag,11:"359670651",]43:7:Expires,29:Mon, 26 Dec 2016 14:25:24 GMT,]50:13:Last-Modified,29:Fri, 09 Aug 2013 23:54:35 GMT,]27:6:Server,14:ECS (iad/18CB),]26:4:Vary,15:Accept-Encoding,]16:7:X-Cache,3:HIT,]25:17:x-ec-custom-error,1:1,]25:14:Content-Length,4:1270,]]}7:request;396:9:is_replay;5:false!17:first_line_format;8:relative;4:port;3:443#7:content;0:,12:stickycookie;5:false!6:method;3:GET,7:headers;82:29:10:User-Agent,11:curl/7.35.0,]26:4:Host,15:www.example.com,]15:6:Accept,3:*/*,]]15:timestamp_start;18:1482157523.9086578^12:http_version;8:HTTP/1.1,13:timestamp_end;18:1482157523.9086578^4:path;1:/,10:stickyauth;5:false!4:host;15:www.example.com,6:scheme;5:https,}7:version;13:1:0#2:18#1:2#]5:error;0:~11:intercepted;5:false!11:server_conn;5143:10:ip_address;56:8:use_ipv6;5:false!7:address;23:13:93.184.216.34;3:443#]}15:timestamp_start;18:1482157523.9086578^19:timestamp_tcp_setup;18:1482157524.0081189^15:ssl_established;4:true!14:source_address;57:8:use_ipv6;5:false!7:address;24:12:10.67.53.133;5:52775#]}19:timestamp_ssl_setup;17:1482157524.260993^4:cert;2122:-----BEGIN CERTIFICATE-----
+7780:6:marked;5:false!11:client_conn;216:19:timestamp_ssl_setup;18:1482157523.9086578^7:address;53:7:address;20:9:127.0.0.1;5:52774#]8:use_ipv6;5:false!}10:clientcert;0:~13:timestamp_end;0:~15:ssl_established;4:true!15:timestamp_start;18:1482157522.8949482^}11:server_conn;5143:3:sni;15:www.example.com;7:address;58:7:address;25:15:www.example.com;3:443#]8:use_ipv6;5:false!}3:via;2570:7:address;58:7:address;25:15:www.example.com;3:443#]8:use_ipv6;5:false!}13:timestamp_end;0:~4:cert;2122:-----BEGIN CERTIFICATE-----
 MIIF8jCCBNqgAwIBAgIQDmTF+8I2reFLFyrrQceMsDANBgkqhkiG9w0BAQsFADBw
 MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
 d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
@@ -82,7 +32,7 @@ ieqRbcuFjmqfyPmUv1U9QoI4TQikpw7TZU0zYZANP4C/gj4Ry48/znmUaRvy2kvI
 l7gRQ21qJTK5suoiYoYNo3J9T+pXPGU7Lydz/HwW+w0DpArtAaukI8aNX4ohFUKS
 wDSiIIWIWJiJGbEeIO0TIFwEVWTOnbNl/faPXpk5IRXicapqiII=
 -----END CERTIFICATE-----
-,13:timestamp_end;0:~3:via;2570:15:ssl_established;4:true!19:timestamp_tcp_setup;18:1482157524.0081189^19:timestamp_ssl_setup;17:1482157524.260993^3:via;0:~3:sni;15:www.example.com;10:ip_address;56:8:use_ipv6;5:false!7:address;23:13:93.184.216.34;3:443#]}15:timestamp_start;18:1482157523.9086578^14:source_address;57:8:use_ipv6;5:false!7:address;24:12:10.67.53.133;5:52775#]}4:cert;2122:-----BEGIN CERTIFICATE-----
+,14:source_address;57:7:address;24:12:10.67.53.133;5:52775#]8:use_ipv6;5:false!}15:timestamp_start;18:1482157523.9086578^10:ip_address;56:7:address;23:13:93.184.216.34;3:443#]8:use_ipv6;5:false!}3:sni;15:www.example.com;3:via;0:~19:timestamp_ssl_setup;17:1482157524.260993^19:timestamp_tcp_setup;18:1482157524.0081189^15:ssl_established;4:true!}13:timestamp_end;0:~4:cert;2122:-----BEGIN CERTIFICATE-----
 MIIF8jCCBNqgAwIBAgIQDmTF+8I2reFLFyrrQceMsDANBgkqhkiG9w0BAQsFADBw
 MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
 d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
@@ -116,4 +66,54 @@ ieqRbcuFjmqfyPmUv1U9QoI4TQikpw7TZU0zYZANP4C/gj4Ry48/znmUaRvy2kvI
 l7gRQ21qJTK5suoiYoYNo3J9T+pXPGU7Lydz/HwW+w0DpArtAaukI8aNX4ohFUKS
 wDSiIIWIWJiJGbEeIO0TIFwEVWTOnbNl/faPXpk5IRXicapqiII=
 -----END CERTIFICATE-----
-,13:timestamp_end;0:~7:address;58:8:use_ipv6;5:false!7:address;25:15:www.example.com;3:443#]}}7:address;58:8:use_ipv6;5:false!7:address;25:15:www.example.com;3:443#]}3:sni;15:www.example.com;}11:client_conn;216:15:timestamp_start;18:1482157522.8949482^15:ssl_established;4:true!13:timestamp_end;0:~10:clientcert;0:~7:address;53:8:use_ipv6;5:false!7:address;20:9:127.0.0.1;5:52774#]}19:timestamp_ssl_setup;18:1482157523.9086578^}6:marked;5:false!}
\ No newline at end of file
+,19:timestamp_ssl_setup;17:1482157524.260993^14:source_address;57:7:address;24:12:10.67.53.133;5:52775#]8:use_ipv6;5:false!}15:ssl_established;4:true!19:timestamp_tcp_setup;18:1482157524.0081189^15:timestamp_start;18:1482157523.9086578^10:ip_address;56:7:address;23:13:93.184.216.34;3:443#]8:use_ipv6;5:false!}}11:intercepted;5:false!5:error;0:~7:version;13:1:0#2:18#1:2#]7:request;396:6:scheme;5:https,4:host;15:www.example.com,10:stickyauth;5:false!4:path;1:/,13:timestamp_end;18:1482157523.9086578^12:http_version;8:HTTP/1.1,15:timestamp_start;18:1482157523.9086578^7:headers;82:29:10:User-Agent,11:curl/7.35.0,]26:4:Host,15:www.example.com,]15:6:Accept,3:*/*,]]6:method;3:GET,12:stickycookie;5:false!7:content;0:,4:port;3:443#17:first_line_format;8:relative;9:is_replay;5:false!}8:response;1815:7:headers;410:25:13:Accept-Ranges,5:bytes,]35:13:Cache-Control,14:max-age=604800,]28:12:Content-Type,9:text/html,]40:4:Date,29:Mon, 19 Dec 2016 14:25:24 GMT,]22:4:Etag,11:"359670651",]43:7:Expires,29:Mon, 26 Dec 2016 14:25:24 GMT,]50:13:Last-Modified,29:Fri, 09 Aug 2013 23:54:35 GMT,]27:6:Server,14:ECS (iad/18CB),]26:4:Vary,15:Accept-Encoding,]16:7:X-Cache,3:HIT,]25:17:x-ec-custom-error,1:1,]25:14:Content-Length,4:1270,]]6:reason;2:OK,11:status_code;3:200#13:timestamp_end;0:~7:content;1270:<!doctype html>
+<html>
+<head>
+    <title>Example Domain</title>
+
+    <meta charset="utf-8" />
+    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <style type="text/css">
+    body {
+        background-color: #f0f0f2;
+        margin: 0;
+        padding: 0;
+        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
+        
+    }
+    div {
+        width: 600px;
+        margin: 5em auto;
+        padding: 50px;
+        background-color: #fff;
+        border-radius: 1em;
+    }
+    a:link, a:visited {
+        color: #38488f;
+        text-decoration: none;
+    }
+    @media (max-width: 700px) {
+        body {
+            background-color: #fff;
+        }
+        div {
+            width: auto;
+            margin: 0 auto;
+            border-radius: 0;
+            padding: 1em;
+        }
+    }
+    </style>    
+</head>
+
+<body>
+<div>
+    <h1>Example Domain</h1>
+    <p>This domain is established to be used for illustrative examples in documents. You may use this
+    domain in examples without prior coordination or asking for permission.</p>
+    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
+</div>
+</body>
+</html>
+,12:http_version;8:HTTP/1.1,15:timestamp_start;0:~}2:id;36:55367415-10f5-4938-b69f-8a523394f947;4:type;4:http;}
\ No newline at end of file