Repository: mitmproxy/mitmproxy

Issue:
Cannot filter by UTF-8 string
##### Steps to reproduce the problem:

1. run `mitmproxy`
2. type (in UTF-8) `: set view_filter=~bs Троцкий`
3. “Invalid interception filter: ~bs Троцкий”
4. type `: set view_filter=~bs trotskij`
5. the filter is set and correctly limits the view to flows where response body contains “trotskij”

##### Any other comments? What have you tried so far?

mitmproxy should make it possible to filter by arbitrary UTF-8 strings.

I have `LANG=en_US.UTF-8`; setting `LC_ALL=en_US.UTF-8` doesn’t help.

These don’t work, either:

- `: set view_filter=~bs "Троцкий"` (the quotes are automatically removed after typing the second one)
- `: set "view_filter=~bs Троцкий"`
- `: set view_filter=~bs /Троцкий/`

##### System information

Mitmproxy: 3.0.3 binary
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-116-generic-x86_64-with-debian-stretch-sid



Discussion:
This might not be an issue of mitmproxy afterall. I think python might not be able to accept the string due to misconfiguration of the terminal emulator.
![deepinscreenshot_select-area_20180304181900](https://user-images.githubusercontent.com/9128903/36945751-94001286-1fd8-11e8-96bf-f8382b43fd3f.png)

![deepinscreenshot_select-area_20180304180343](https://user-images.githubusercontent.com/9128903/36945685-c25ddeb6-1fd7-11e8-8083-3bb430258ada.png)

Thanks for reporting this! refs #2801 

Resolving patch:
diff --git a/mitmproxy/flowfilter.py b/mitmproxy/flowfilter.py
--- a/mitmproxy/flowfilter.py
+++ b/mitmproxy/flowfilter.py
@@ -475,7 +475,30 @@ def _make():
         parts.append(f)
 
     simplerex = "".join(c for c in pp.printables if c not in "()~'\"")
+    alphdevanagari = pp.pyparsing_unicode.Devanagari.alphas
+    alphcyrillic = pp.pyparsing_unicode.Cyrillic.alphas
+    alphgreek = pp.pyparsing_unicode.Greek.alphas
+    alphchinese = pp.pyparsing_unicode.Chinese.alphas
+    alpharabic = pp.pyparsing_unicode.Arabic.alphas
+    alphhebrew = pp.pyparsing_unicode.Hebrew.alphas
+    alphjapanese = pp.pyparsing_unicode.Japanese.alphas
+    alphkorean = pp.pyparsing_unicode.Korean.alphas
+    alphlatin1 = pp.pyparsing_unicode.Latin1.alphas
+    alphlatinA = pp.pyparsing_unicode.LatinA.alphas
+    alphlatinB = pp.pyparsing_unicode.LatinB.alphas
+
     rex = pp.Word(simplerex) |\
+        pp.Word(alphcyrillic) |\
+        pp.Word(alphgreek) |\
+        pp.Word(alphchinese) |\
+        pp.Word(alpharabic) |\
+        pp.Word(alphdevanagari) |\
+        pp.Word(alphhebrew) |\
+        pp.Word(alphjapanese) |\
+        pp.Word(alphkorean) |\
+        pp.Word(alphlatin1) |\
+        pp.Word(alphlatinA) |\
+        pp.Word(alphlatinB) |\
         pp.QuotedString("\"", escChar='\\') |\
         pp.QuotedString("'", escChar='\\')
     for klass in filter_rex:


Test patch:
diff --git a/test/mitmproxy/test_flowfilter.py b/test/mitmproxy/test_flowfilter.py
--- a/test/mitmproxy/test_flowfilter.py
+++ b/test/mitmproxy/test_flowfilter.py
@@ -28,6 +28,9 @@ def test_simple(self):
         self._dump(p)
         assert len(p.lst) == 2
 
+    def test_non_ascii(self):
+        assert flowfilter.parse("~s шгн")
+
     def test_naked_url(self):
         a = flowfilter.parse("foobar ~h rex")
         assert a.lst[0].expr == "foobar"
@@ -173,10 +176,30 @@ def match_body(self, q, s):
         assert not self.q("~bq message", q)
         assert not self.q("~bq message", s)
 
+        s.response.text = 'яч'           # Cyrillic
+        assert self.q("~bs яч", s)
+        s.response.text = '测试'          # Chinese
+        assert self.q('~bs 测试', s)
+        s.response.text = 'ॐ'            # Hindi
+        assert self.q('~bs ॐ', s)
+        s.response.text = 'لله'           # Arabic
+        assert self.q('~bs لله', s)
+        s.response.text = 'θεός'          # Greek
+        assert self.q('~bs θεός', s)
+        s.response.text = 'לוהים'          # Hebrew
+        assert self.q('~bs לוהים', s)
+        s.response.text = '神'            # Japanese
+        assert self.q('~bs 神', s)
+        s.response.text = '하나님'         # Korean
+        assert self.q('~bs 하나님', s)
+        s.response.text = 'Äÿ'            # Latin
+        assert self.q('~bs Äÿ', s)
+
         assert not self.q("~bs nomatch", s)
         assert not self.q("~bs content", q)
         assert not self.q("~bs content", s)
         assert not self.q("~bs message", q)
+        s.response.text = 'message'
         assert self.q("~bs message", s)
 
     def test_body(self):