Repository: mitmproxy/mitmproxy

Issue:
Setting View Filter shows Addon Error in Eventlog
    error: Addon error: Traceback (most recent call last):
        File "/home/coldfire/Projects/mitmproxy/mitmproxy/addons/view.py", line 521, in update
            idx = self._view.index(f)
        File "/home/coldfire/Projects/mitmproxy/venv/lib/python3.6/site-packages/sortedcontainers/sortedlist.py", line 2343, in index
            raise ValueError('{0!r} is not in list'.format(val))
        File "/home/coldfire/Projects/mitmproxy/mitmproxy/http.py", line 182, in __repr__
            return s.format(flow=self)
        File "/home/coldfire/Projects/mitmproxy/mitmproxy/connections.py", line 215, in __repr__
            host=self.address[0],
    TypeError: 'NoneType' object is not subscriptable


##### Steps to reproduce the problem:
1. Open `mitmproxy`
2.  set a value for `view_filter` in Options or using the ` : set view_filter={str} `
3. Open a attached browser and open any url
4. Quit the browser and Open the Eventlog in mitmproxy. 


##### Any other comments? What have you tried so far?
Another error occuring for the view_filter option. I was working on a fix for #2801 #2951 when i noticed this other error occurs if any value is setup in view_filter. I think it requires a complete patch once and for all. 

##### System information
Mitmproxy: 4.0.0.dev103 (commit 77ed33b) 
Python:    3.6.4
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.14.21-1-MANJARO-x86_64-with-arch-Manjaro-Linux



Discussion:
I noticed a weird thing, adding `time.sleep(1)` just before the return statement fixes the problem and no errors are generated. I am not sure why or how is this happening. 
https://github.com/mitmproxy/mitmproxy/blob/cc00b03554d84a37b55f911a84d70b6d7909c9cb/mitmproxy/connections.py#L212  
I'm seeing this too. The error appears onscreen and makes the feature hard to use. Is there a way to hide the error from displaying until a fix is found?
Not sure why we have a connection with `address = None` at this point, but in any case we should make `ClientConnection.__repr__` and `ServerConnection.__repr__` robust so that they don't crash in this case. This can be either fixed in those functions directly, or we amend `mitmproxy.util.human.format_address` to also accept `None` and then return `<no address information>` or something alike - both works for me. If someone fixes this, please add a test so that it doesn't happen again! 😃 
@mhils  I think there is another way, we can handle this. As noticed earlier, if we add `time.sleep(1)` before the return statement, somehow the self.address  is set to a actual address value. So instead of handling self.address for `None` , can we check whether self.address is `None` and if it is  give it some time set an actual value.

    if self.address is None:
            time.sleep(1)
    return {str} 
 
@fenilgandhi I have an assumption concerning this issue. I am not sure it is absolutely correct.
As you can see the issue appears when mitmproxy tries to give us a representation of `ServerConnection` object, when `ValueError` is raised inside https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/addons/view.py#L521

For some reasons for some flows `self.address` is still `None`, when we need it to be available in `ServerConnection`. When we write `time.sleep(1)`, we suspend execution of the calling thread for 1 second. **Perhaps** mitmproxy switches to other available thread and this thread set the value for `self.address`. The problem is that suspended thread is `MainThread` and every new case with `self.address is None` almost all mitmproxy functions will stop for 1 second.
It is also possible, that for some reasons mitmproxy gets `self.address` with a delay and when we write `time.sleep(1)` we compensate this delay. 

btw `time.sleep(0.1)` works as well.

@kajojify  I had the same notion but since i could not  find the setter/thread  which sets the value during the sleep, so i was not sure how to proceed with this. But if what you are saying is true, we can make sure that the setter method sets a value before the ServerConnection.__repr__  is returned. 

setting `time.sleep(1)` was just a first instinct, but yeah `time.sleep(0.1)` works well too and if we want a fast temporary fix , we can use it.
I can't reproduce this. Here's what I'm doing:

1. Start `mitmproxy`
2. `:set view_filter=foo`
3. switch to second window, enter `curl -x localhost:8080 -k https://google.com/`
4. Press `E` to open the event log.
@mhils I reproduced it only using a browser.

Resolving patch:
diff --git a/mitmproxy/connections.py b/mitmproxy/connections.py
--- a/mitmproxy/connections.py
+++ b/mitmproxy/connections.py
@@ -1,18 +1,18 @@
-import time
-
 import os
+import time
 import typing
 import uuid
 
-from mitmproxy import stateobject, exceptions
 from mitmproxy import certs
+from mitmproxy import exceptions
+from mitmproxy import stateobject
 from mitmproxy.net import tcp
 from mitmproxy.net import tls
+from mitmproxy.utils import human
 from mitmproxy.utils import strutils
 
 
 class ClientConnection(tcp.BaseHandler, stateobject.StateObject):
-
     """
     A client connection
 
@@ -72,11 +72,10 @@ def __repr__(self):
         else:
             alpn = ""
 
-        return "<ClientConnection: {tls}{alpn}{host}:{port}>".format(
+        return "<ClientConnection: {tls}{alpn}{address}>".format(
             tls=tls,
             alpn=alpn,
-            host=self.address[0],
-            port=self.address[1],
+            address=human.format_address(self.address),
         )
 
     def __eq__(self, other):
@@ -161,7 +160,6 @@ def finish(self):
 
 
 class ServerConnection(tcp.TCPClient, stateobject.StateObject):
-
     """
     A server connection
 
@@ -209,11 +207,10 @@ def __repr__(self):
             )
         else:
             alpn = ""
-        return "<ServerConnection: {tls}{alpn}{host}:{port}>".format(
+        return "<ServerConnection: {tls}{alpn}{address}>".format(
             tls=tls,
             alpn=alpn,
-            host=self.address[0],
-            port=self.address[1],
+            address=human.format_address(self.address),
         )
 
     def __eq__(self, other):
diff --git a/mitmproxy/utils/human.py b/mitmproxy/utils/human.py
--- a/mitmproxy/utils/human.py
+++ b/mitmproxy/utils/human.py
@@ -73,11 +73,13 @@ def format_timestamp_with_milli(s):
     return d.strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
 
 
-def format_address(address: tuple) -> str:
+def format_address(address: typing.Optional[tuple]) -> str:
     """
     This function accepts IPv4/IPv6 tuples and
     returns the formatted address string with port number
     """
+    if address is None:
+        return "<no address>"
     try:
         host = ipaddress.ip_address(address[0])
         if host.is_unspecified:


Test patch:
diff --git a/test/mitmproxy/test_connections.py b/test/mitmproxy/test_connections.py
--- a/test/mitmproxy/test_connections.py
+++ b/test/mitmproxy/test_connections.py
@@ -38,6 +38,9 @@ def test_repr(self):
         assert 'ALPN' not in repr(c)
         assert 'TLS' in repr(c)
 
+        c.address = None
+        assert repr(c)
+
     def test_tls_established_property(self):
         c = tflow.tclient_conn()
         c.tls_established = True
@@ -110,6 +113,9 @@ def test_repr(self):
         c.tls_established = False
         assert 'TLS' not in repr(c)
 
+        c.address = None
+        assert repr(c)
+
     def test_tls_established_property(self):
         c = tflow.tserver_conn()
         c.tls_established = True
diff --git a/test/mitmproxy/utils/test_human.py b/test/mitmproxy/utils/test_human.py
--- a/test/mitmproxy/utils/test_human.py
+++ b/test/mitmproxy/utils/test_human.py
@@ -56,3 +56,4 @@ def test_format_address():
     assert human.format_address(("example.com", "54010")) == "example.com:54010"
     assert human.format_address(("::", "8080")) == "*:8080"
     assert human.format_address(("0.0.0.0", "8080")) == "*:8080"
+    assert human.format_address(None) == "<no address>"