Repository: mitmproxy/mitmproxy

Issue:
No way to set a sequence option on the command line
##### Steps to reproduce the problem:

1. Create a file `/tmp/example.py` with the following text:
```python
import typing
from mitmproxy import ctx

class Example:

    def load(self, loader):
        loader.add_option(name='myoption', typespec=typing.Sequence[str],
                          default=[], help='')

    def response(self, flow):
        ctx.log.info(repr(ctx.options.myoption))

addons = [Example()]
```
2. Run `mitmdump -s /tmp/example.py --set myoption=foo --set myoption=bar`
3. Run a request through it: `curl -sx http://localhost:8080 http://httpbin.org/get`
4. Observe that `['bar']` is printed, meaning that `myoption` is set to a sequence of one element — the last one passed on the command line, with the first one ignored.


##### Any other comments? What have you tried so far?

Looking at [the code](https://github.com/mitmproxy/mitmproxy/blob/79fc4e8ad6cab5181ba39d7f0b0d40f7c11a8389/mitmproxy/optmanager.py#L312-L316), I’m not immediately sure why the multiple `--set` don’t work as I expected them to.

##### System information

Mitmproxy: 4.0.0 (79fc4e8)
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0h  27 Mar 2018
Platform:  Linux-4.4.0-116-generic-x86_64-with-Ubuntu-16.04-xenial



Discussion:
Thanks for the concise example, this looks like a bug!
I will take a look at this :)
To solve this properly, we will have to define a format for sequences of strings, and teach the `parse_setval` function to know about it. @madt1m mad a valiant attempt to avoid doing this, but I don't think it works. ;)
@cortesi could something in line with the good ol' $PATH-like format do the trick? 
--set myopt="opt1;opt2;..."  
I am using semicolon, because i guess that colon could be part of a string. Or, maybe, give the user the possibility to define its own delimiter?
That's tricky because it's plausible to have any of those delimiters in e.g. a regex pattern. I'm afraid we have to go the strenuous route and add an additional "temp dict" argument to `parse_setval`.
I don't want there to be any ambiguity about set on the command-line or in the command prompt: if you set an option, you over-write what was there before in toto. So no book-keeping between --set options, please.

I suggest we do two things:

- Teach `--set` to know about delimited strings. I think `;` or `,` are both fine, and add backslash escaping. This gets complicated in the presence of a shell, but users are familiar with this from all sorts of contexts, and it's bearable.
- I'd be happy to have an `--append` command-line option and matching command that appends a value to a sequence option. This can be used where we don't mind if update is called after every append. We also might need a better name than "append" which is a bit over-loaded.





I'll manage to allocate some time in this weekend for this. I'll try to come up with a feasible solution :)
Just to chime in: This bug / limitation has bit me, as well. I'm going to have to use a string option instead with a custom delimiter and hand-parse it myself to upgrade from MITMProxy 2 (where I just used argparse) to 4.
I'm having the same issue, but with `server_replay_use_headers`. There doesn't seem to be a command-line argument to add such headers and using `--set server_replay_use_headers=headera,headerb` or `--set server_replay_use_headers=headera --set server_replay_use_headers=headerb` doesn't work correctly.
Is there any way to circumvent this limitation? I tried adding an inline script to do something, but it doesn't seem to work.

I.e.
```
from mitmproxy import ctx

ctx.options.server_replay_ignore_params = [
    "one",
    "two"
]
```

It looks correct in `mitmproxy`, but it does not seem to actually do anything.
<img width="371" alt="image" src="https://user-images.githubusercontent.com/641248/93106496-a0c28100-f6b0-11ea-8806-1fe60f91cd30.png">


Resolving patch:
diff --git a/mitmproxy/addons/core.py b/mitmproxy/addons/core.py
--- a/mitmproxy/addons/core.py
+++ b/mitmproxy/addons/core.py
@@ -56,6 +56,8 @@ def set(self, option: str, value: str = "") -> None:
             strings and integers are set to None (if permitted), and sequences
             are emptied. Boolean values can be true, false or toggle.
             Multiple values are concatenated with a single space.
+            Sequences are set using multiple invocations to set for
+            the same option.
         """
         strspec = f"{option}={value}"
         try:
diff --git a/mitmproxy/optmanager.py b/mitmproxy/optmanager.py
--- a/mitmproxy/optmanager.py
+++ b/mitmproxy/optmanager.py
@@ -1,3 +1,4 @@
+import collections
 import contextlib
 import blinker
 import blinker._saferef
@@ -91,7 +92,7 @@ class OptManager:
         mutation doesn't change the option state inadvertently.
     """
     def __init__(self):
-        self.deferred: typing.Dict[str, str] = {}
+        self.deferred: typing.Dict[str, typing.List[str]] = {}
         self.changed = blinker.Signal()
         self.errored = blinker.Signal()
         # Options must be the last attribute here - after that, we raise an
@@ -295,7 +296,7 @@ def set(self, *spec, defer=False):
             are added.
         """
         vals = {}
-        unknown = {}
+        unknown: typing.Dict[str, typing.List[str]] = collections.defaultdict(list)
         for i in spec:
             parts = i.split("=", maxsplit=1)
             if len(parts) == 1:
@@ -303,9 +304,9 @@ def set(self, *spec, defer=False):
             else:
                 optname, optval = parts[0], parts[1]
             if optname in self._options:
-                vals[optname] = self.parse_setval(self._options[optname], optval)
+                vals[optname] = self.parse_setval(self._options[optname], optval, vals.get(optname))
             else:
-                unknown[optname] = optval
+                unknown[optname].append(optval)
         if defer:
             self.deferred.update(unknown)
         elif unknown:
@@ -318,15 +319,16 @@ def process_deferred(self):
             have since been added.
         """
         update = {}
-        for optname, optval in self.deferred.items():
+        for optname, optvals in self.deferred.items():
             if optname in self._options:
-                optval = self.parse_setval(self._options[optname], optval)
-                update[optname] = optval
+                for optval in optvals:
+                    optval = self.parse_setval(self._options[optname], optval, update.get(optname))
+                    update[optname] = optval
         self.update(**update)
         for k in update.keys():
             del self.deferred[k]
 
-    def parse_setval(self, o: _Option, optstr: typing.Optional[str]) -> typing.Any:
+    def parse_setval(self, o: _Option, optstr: typing.Optional[str], currentvalue: typing.Any) -> typing.Any:
         """
             Convert a string to a value appropriate for the option type.
         """
@@ -357,7 +359,10 @@ def parse_setval(self, o: _Option, optstr: typing.Optional[str]) -> typing.Any:
             if not optstr:
                 return []
             else:
-                return getattr(self, o.name) + [optstr]
+                if currentvalue:
+                    return getattr(self, o.name) + currentvalue + [optstr]
+                else:
+                    return getattr(self, o.name) + [optstr]
         raise NotImplementedError("Unsupported option type: %s", o.typespec)
 
     def make_parser(self, parser, optname, metavar=None, short=None):
diff --git a/mitmproxy/tools/cmdline.py b/mitmproxy/tools/cmdline.py
--- a/mitmproxy/tools/cmdline.py
+++ b/mitmproxy/tools/cmdline.py
@@ -27,6 +27,8 @@ def common_options(parser, opts):
             Set an option. When the value is omitted, booleans are set to true,
             strings and integers are set to None (if permitted), and sequences
             are emptied. Boolean values can be true, false or toggle.
+            Sequences are set using multiple invocations to set for
+            the same option.
         """
     )
     parser.add_argument(


Test patch:
diff --git a/test/mitmproxy/test_optmanager.py b/test/mitmproxy/test_optmanager.py
--- a/test/mitmproxy/test_optmanager.py
+++ b/test/mitmproxy/test_optmanager.py
@@ -71,7 +71,7 @@ def test_defaults():
 def test_required_int():
     o = TO()
     with pytest.raises(exceptions.OptionsError):
-        o.parse_setval(o._options["required_int"], None)
+        o.parse_setval(o._options["required_int"], None, None)
 
 
 def test_deepcopy():
@@ -439,6 +439,9 @@ def test_set():
     opts.set("seqstr")
     assert opts.seqstr == []
 
+    opts.set(*('seqstr=foo', 'seqstr=bar'))
+    assert opts.seqstr == ["foo", "bar"]
+
     with pytest.raises(exceptions.OptionsError):
         opts.set("deferredoption=wobble")
 
@@ -450,3 +453,12 @@ def test_set():
     opts.process_deferred()
     assert "deferredoption" not in opts.deferred
     assert opts.deferredoption == "wobble"
+
+    opts.set(*('deferredsequenceoption=a', 'deferredsequenceoption=b'), defer=True)
+    assert "deferredsequenceoption" in opts.deferred
+    opts.process_deferred()
+    assert "deferredsequenceoption" in opts.deferred
+    opts.add_option("deferredsequenceoption", typing.Sequence[str], [], "help")
+    opts.process_deferred()
+    assert "deferredsequenceoption" not in opts.deferred
+    assert opts.deferredsequenceoption == ["a", "b"]