Repository: mitmproxy/mitmproxy

Issue:
How to configure file system watchdog (away) for attached scripts
Hi folks,

I upgraded my local mitmproxy using pip from 0.17 to 0.18.2 and now my script does not work anymore. The mitmproxy is used to connect a local machine with a corporate proxy and measure the traffic in between. The mitmproxy (to be exact, the mitmdump program) is called like this:

    /usr/local/bin/mitmdump -p 9090 -U 'http://proxy:8080' -s 'tracer.py argument'

The `tracer.py` has three functions implemented as described in the [Events Documentation](http://docs.mitmproxy.org/en/latest/scripting/events.html):

    def start():
        # should be called once on startup
        # so do some initialization here
    
    def response(flow):
        # track the response information
    
    def done():
        # called at script shutdown

In the stdout I see a lot of lines telling me `Reloading script: tracer.py` and if I add a print('hello from start()') to the `start()` function I also see this message in stdout, which means that `start()` gets executed more than once. My search for `Reloading script` pointed me to [the function `tick` in addons/script.py](https://github.com/mitmproxy/mitmproxy/blob/6792cc1de90b02b89da9ae80704ea237c5486e66/mitmproxy/addons/script.py#L185), which seems to be running in a certain interval. The problem I can address so far here is that what happens when `self.should_reload.is_set()` returns `True` while the proxy is running.

`should_reload`s internal state is set to `False` by `clear()` in line 187, but it could be set to `True` again by `reload()` in line 175. `reload` is passed as callback to a `ReloadHandler` in line 204, which is connected to a file system watchdog. This sounds similar to the [Scripting Overview](http://docs.mitmproxy.org/en/latest/scripting/overview.html#developing-scripts),

> Mitmprxoy monitors scripts for modifications, and reloads them on change. When this happens, the script is shut down (the done event is called), and the new instance is started up as if the script had just been loaded (the start and configure events are called).

with the exception, that the `ReloadHandler` reloads the script if _anything_ in the observed directory is changed. So I propose not only to filter directories and files starting with a `.`, but to watch _only_ for files ending with `.py`.

##### System information

The output of "mitmdump --sysinfo":

Mitmproxy version: 0.18.2
Python version: 2.7.12
Platform: Linux-4.4.0-42-generic-x86_64-with-LinuxMint-18-sarah
SSL version: OpenSSL 1.0.2g  1 Mar 2016
Linux distro: LinuxMint 18 sarah
How to configure file system watchdog (away) for attached scripts
Hi folks,

I upgraded my local mitmproxy using pip from 0.17 to 0.18.2 and now my script does not work anymore. The mitmproxy is used to connect a local machine with a corporate proxy and measure the traffic in between. The mitmproxy (to be exact, the mitmdump program) is called like this:

    /usr/local/bin/mitmdump -p 9090 -U 'http://proxy:8080' -s 'tracer.py argument'

The `tracer.py` has three functions implemented as described in the [Events Documentation](http://docs.mitmproxy.org/en/latest/scripting/events.html):

    def start():
        # should be called once on startup
        # so do some initialization here
    
    def response(flow):
        # track the response information
    
    def done():
        # called at script shutdown

In the stdout I see a lot of lines telling me `Reloading script: tracer.py` and if I add a print('hello from start()') to the `start()` function I also see this message in stdout, which means that `start()` gets executed more than once. My search for `Reloading script` pointed me to [the function `tick` in addons/script.py](https://github.com/mitmproxy/mitmproxy/blob/6792cc1de90b02b89da9ae80704ea237c5486e66/mitmproxy/addons/script.py#L185), which seems to be running in a certain interval. The problem I can address so far here is that what happens when `self.should_reload.is_set()` returns `True` while the proxy is running.

`should_reload`s internal state is set to `False` by `clear()` in line 187, but it could be set to `True` again by `reload()` in line 175. `reload` is passed as callback to a `ReloadHandler` in line 204, which is connected to a file system watchdog. This sounds similar to the [Scripting Overview](http://docs.mitmproxy.org/en/latest/scripting/overview.html#developing-scripts),

> Mitmprxoy monitors scripts for modifications, and reloads them on change. When this happens, the script is shut down (the done event is called), and the new instance is started up as if the script had just been loaded (the start and configure events are called).

with the exception, that the `ReloadHandler` reloads the script if _anything_ in the observed directory is changed. So I propose not only to filter directories and files starting with a `.`, but to watch _only_ for files ending with `.py`.

##### System information

The output of "mitmdump --sysinfo":

Mitmproxy version: 0.18.2
Python version: 2.7.12
Platform: Linux-4.4.0-42-generic-x86_64-with-LinuxMint-18-sarah
SSL version: OpenSSL 1.0.2g  1 Mar 2016
Linux distro: LinuxMint 18 sarah


Discussion:
Thanks for the report!

While there are cases where always-reload is useful, only reloading on *.py files seems to be a reasonable default to me. Judging from bug reports, this seems to be a common annoyance right now. If we want to have the always-reload functioanlity again, we could make that configurable at some point.

@cortesi, IIRC you championed the "always-reload" paradigm - would you be okay with having *.py-reload only by default?


I think we have to accept that we can never cater perfectly for all use-cases here, and that someone is always going to be surprised whatever behaviour is chosen. I am OK with either strategy, as long as it's clearly documented.

I also have a strawman proposal - maybe mitmproxy shouldn't be doing this at all. Instead, we can provide a script-reload function through a web API, and let users use a tool like [modd](https://github.com/cortesi/modd) instead to get precisely the behaviour they expect. This would also remove one of our trickier dependencies. 




The idea of auto-reloading is not so bad (you can reload by simply `touch`ing), but the current implementation is somehow surprising, as it observes the whole directory, while the documentation only mentions the actual script.
@RobbBienert: If @cortesi is fine with either behaviour, I think we'd be more than happy to accept a PR that changes the implementation (and updates the docs)! :smiley:

You've probably seen this already, the place to fix this is [`mitmproxy/addons/script.py#L124`](https://github.com/mitmproxy/mitmproxy/blob/6792cc1de90b02b89da9ae80704ea237c5486e66/mitmproxy/addons/script.py#L124)! 😊 
I'm back again and working on the fix -- see pull request #1911 
Thanks for the report!

While there are cases where always-reload is useful, only reloading on *.py files seems to be a reasonable default to me. Judging from bug reports, this seems to be a common annoyance right now. If we want to have the always-reload functioanlity again, we could make that configurable at some point.

@cortesi, IIRC you championed the "always-reload" paradigm - would you be okay with having *.py-reload only by default?


I think we have to accept that we can never cater perfectly for all use-cases here, and that someone is always going to be surprised whatever behaviour is chosen. I am OK with either strategy, as long as it's clearly documented.

I also have a strawman proposal - maybe mitmproxy shouldn't be doing this at all. Instead, we can provide a script-reload function through a web API, and let users use a tool like [modd](https://github.com/cortesi/modd) instead to get precisely the behaviour they expect. This would also remove one of our trickier dependencies. 




The idea of auto-reloading is not so bad (you can reload by simply `touch`ing), but the current implementation is somehow surprising, as it observes the whole directory, while the documentation only mentions the actual script.
@RobbBienert: If @cortesi is fine with either behaviour, I think we'd be more than happy to accept a PR that changes the implementation (and updates the docs)! :smiley:

You've probably seen this already, the place to fix this is [`mitmproxy/addons/script.py#L124`](https://github.com/mitmproxy/mitmproxy/blob/6792cc1de90b02b89da9ae80704ea237c5486e66/mitmproxy/addons/script.py#L124)! 😊 
I'm back again and working on the fix -- see pull request #1911 

Resolving patch:
diff --git a/mitmproxy/addons/script.py b/mitmproxy/addons/script.py
--- a/mitmproxy/addons/script.py
+++ b/mitmproxy/addons/script.py
@@ -110,11 +110,16 @@ def __init__(self, callback):
         self.callback = callback
 
     def filter(self, event):
+        """
+            Returns True only when .py file is changed
+        """
         if event.is_directory:
             return False
         if os.path.basename(event.src_path).startswith("."):
             return False
-        return True
+        if event.src_path.endswith(".py"):
+            return True
+        return False
 
     def on_modified(self, event):
         if self.filter(event):


Test patch:
diff --git a/test/mitmproxy/addons/test_script.py b/test/mitmproxy/addons/test_script.py
--- a/test/mitmproxy/addons/test_script.py
+++ b/test/mitmproxy/addons/test_script.py
@@ -44,14 +44,19 @@ def test_reloadhandler():
     rh = script.ReloadHandler(Called())
     assert not rh.filter(watchdog.events.DirCreatedEvent("path"))
     assert not rh.filter(watchdog.events.FileModifiedEvent("/foo/.bar"))
-    assert rh.filter(watchdog.events.FileModifiedEvent("/foo/bar"))
+    assert not rh.filter(watchdog.events.FileModifiedEvent("/foo/bar"))
+    assert rh.filter(watchdog.events.FileModifiedEvent("/foo/bar.py"))
 
     assert not rh.callback.called
     rh.on_modified(watchdog.events.FileModifiedEvent("/foo/bar"))
+    assert not rh.callback.called
+    rh.on_modified(watchdog.events.FileModifiedEvent("/foo/bar.py"))
     assert rh.callback.called
     rh.callback.called = False
 
     rh.on_created(watchdog.events.FileCreatedEvent("foo"))
+    assert not rh.callback.called
+    rh.on_created(watchdog.events.FileCreatedEvent("foo.py"))
     assert rh.callback.called