internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: data-for-change/anyway

Issue:
Localize Widget - accidents_heat_map
**Widget ID**
accidents_heat_map

**Widget Fields**
only title

**Dictionaries Missing**
use segment dictionary used in [this pr](https://github.com/hasadna/anyway/pull/1590/files)
Note: Create a dictionary containing using the DB code as a key and not the Hebrew translation (for example `{1: 'fatal'}` rather than `{'קטלנית': 'fatal'}`)

**Widget Title**
Yes, current title: מוקדי תאונות קטלניות וקשות במקטע ___

**Additional context**
- Existing translation to CBS fields - [see here](https://docs.google.com/spreadsheets/d/1gjeMsPEWayMZ4Z0tGDKkL0mDsXgIRQ4-aAadRpEP9Oc/edit?usp=sharing)
- Existing CBS Fields - [see here](https://docs.google.com/spreadsheets/d/1qaVV7NKXVYNmnxKZ4he2MKZDAjWPHiHfq-U5dcNZM5k/edit?usp=sharing)
(Note - if translation in file is not adequate - please create a better translation)

**Flask Babel**
For updating messages.pot with all strings for translation - use the following command: `pybabel extract -F babel.cfg -o messages.pot .`
For updating existing po files with new strings: `pybabel update -i messages.pot -d translations`



Discussion:

Resolving patch:
diff --git a/anyway/infographics_utils.py b/anyway/infographics_utils.py
--- a/anyway/infographics_utils.py
+++ b/anyway/infographics_utils.py
@@ -478,13 +478,11 @@ def get_accident_count_by_accident_type(location_info, start_time, end_time):
 @register
 class AccidentsHeatMapWidget(Widget):
     name: str = "accidents_heat_map"
+    BASE_TITLE = _("Fatal and severe accidents heat map")
 
     def __init__(self, request_params: RequestParams):
         super().__init__(request_params, type(self).name)
         self.rank = 7
-        self.text = {
-            "title": AccidentsHeatMapWidget.get_heat_map_title(request_params.location_info)
-        }
 
     def generate_items(self) -> None:
         accidents_heat_map_filters = self.request_params.location_info.copy()
@@ -498,10 +496,6 @@ def generate_items(self) -> None:
             end_time=self.request_params.end_time,
         )
 
-    @staticmethod
-    def get_heat_map_title(location_info):
-        return "מוקדי תאונות קטלניות וקשות במקטע " + location_info["road_segment_name"]
-
     @staticmethod
     def get_accidents_heat_map(filters, start_time, end_time):
         filters = filters or {}
@@ -514,6 +508,14 @@ def get_accidents_heat_map(filters, start_time, end_time):
         df = pd.read_sql_query(query.statement, query.session.bind)
         return df.to_dict(orient="records")  # pylint: disable=no-member
 
+    @staticmethod
+    def localize_items(request_params: RequestParams, items: Dict) -> Dict:
+        items["data"]["text"] = {
+            "title": _(AccidentsHeatMapWidget.BASE_TITLE)
+            + " "
+            + segment_dictionary[request_params.location_info["road_segment_name"]]
+        }
+        return items
 
 @register
 class AccidentCountByAccidentYearWidget(Widget):


Test patch:
Repository: data-for-change/anyway

Issue:
Localization after cache does not work in  localization in accident_count_by_accident_year and injured_count_by_accident_year[Bug]
Localization after cache does not work in  localization in accident_count_by_accident_year and injured_count_by_accident_year[Bug]

**To Reproduce**
Steps to reproduce the behavior:
1.Run: www.anyway.co.il/api/infographics-data?news_flash_id=16577&years_ago=5&lang=he
2. `localized_name` remains in English
**Expected behavior**
Should be in Hebrew



Discussion:

Resolving patch:
diff --git a/anyway/infographics_utils.py b/anyway/infographics_utils.py
--- a/anyway/infographics_utils.py
+++ b/anyway/infographics_utils.py
@@ -592,8 +592,7 @@ def __init__(self, request_params: RequestParams):
         super().__init__(request_params, type(self).name)
         self.rank = 8
         self.text = {
-            # "title" will be set in localize_items()
-            "labels": gen_entity_labels(AccidentSeverity)
+            # "title" and "labels" will be set in localize_items()
         }
         self.information = "Fatal, severe and light accidents count in the specified years, split by accident severity"
 
@@ -619,10 +618,12 @@ def generate_items(self) -> None:
 
     @staticmethod
     def localize_items(request_params: RequestParams, items: Dict) -> Dict:
-        items["data"]["text"]["title"] =\
-            _("Number of accidents, by year, splitted by accident severity, in segment")\
-            + " "\
-            + segment_dictionary[request_params.location_info["road_segment_name"]]
+        items["data"]["text"] = {
+            "title": _("Number of accidents, by year, splitted by accident severity, in segment")
+                     + " "
+                     + segment_dictionary[request_params.location_info["road_segment_name"]],
+            "labels": gen_entity_labels(AccidentSeverity)
+        }
         return items
 
 
@@ -634,8 +635,7 @@ def __init__(self, request_params: RequestParams):
         super().__init__(request_params, type(self).name)
         self.rank = 9
         self.text = {
-            # "title" will be set in localize_items()
-            "labels": gen_entity_labels(InjurySeverity)
+            # "title" and "labels" will be set in localize_items()
         }
         self.information = "Fatal, severe and light injured count in the specified years, split by injury severity"
 
@@ -660,10 +660,12 @@ def generate_items(self) -> None:
 
     @staticmethod
     def localize_items(request_params: RequestParams, items: Dict) -> Dict:
-        items["data"]["text"]["title"] =\
-            _("Number of injured in accidents, per year, splitted by severity, in segment")\
-            + " "\
-            + segment_dictionary[request_params.location_info["road_segment_name"]]
+        items["data"]["text"] = {
+            "title": _("Number of injured in accidents, per year, splitted by severity, in segment")
+                     + " "
+                     + segment_dictionary[request_params.location_info["road_segment_name"]],
+            "labels": gen_entity_labels(InjurySeverity)
+        }
         return items
 
 


Test patch:
Repository: data-for-change/anyway

Issue:
Open311 - Check what can be used 
<!---
@huboard:{"order":7.8125}
-->



Discussion:
Some Open311 info and links I gathered:

It is a **specification** of an API for reporting geo (non-critical) issues

Specifically the latest Geo spec: http://wiki.open311.org/GeoReport_v2

List of projects: http://wiki.open311.org/Open_Source

Of possible interest:
Simple Flask Server: https://github.com/miamidade/georeport-server
http://www.ushahidi.com/
    emergency oriented, even sms clients, php mysql, localization includes Hebrew
    crowdmap.com variant for non-technical, seems without geospatial
http://en.seeclickfix.com/
- http://en.seeclickfix.com/chicago
  Facebook App: https://github.com/codeforamerica/open311_facebook, 
  http://stanfordrosenthal.com/open311/

Other based platform: Fix-my-street, Mark-a-spot


Resolving patch:
diff --git a/anyway/infographics_utils.py b/anyway/infographics_utils.py
--- a/anyway/infographics_utils.py
+++ b/anyway/infographics_utils.py
@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+import time
 import logging
 import datetime
 import json
@@ -1852,12 +1853,13 @@ def create_infographics_data(news_flash_id, number_of_years_ago, lang: str) -> s
     return json.dumps(output, default=str)
 
 
-# def create_infographics_data_1(request_params: RequestParams) -> str:
-#     output = create_infographics_items(request_params)
-#     return json.dumps(output, default=str)
-
-
 def create_infographics_items(request_params: RequestParams) -> Dict:
+    def get_dates_comment():
+        return {
+            "date_range": [request_params.start_time.year, request_params.end_time.year],
+            "last_update": time.mktime(request_params.end_time.timetuple())
+        }
+
     try:
         if request_params is None:
             return {}
@@ -1874,15 +1876,9 @@ def create_infographics_items(request_params: RequestParams) -> Dict:
         output["meta"] = {
             "location_info": request_params.location_info.copy(),
             "location_text": request_params.location_text,
+            "dates_comment": get_dates_comment()
         }
         output["widgets"] = []
-        output["meta"]["dates_comment"] = (
-            str(request_params.start_time.year)
-            + "-"
-            + str(request_params.end_time.year)
-            + ", עדכון אחרון: "
-            + str(request_params.end_time)
-        )
         widgets: List[Widget] = generate_widgets(request_params=request_params, to_cache=True)
         widgets.extend(generate_widgets(request_params=request_params, to_cache=False))
         output["widgets"].extend(list(map(lambda w: w.serialize(), widgets)))


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Replay behaviour in non-live reverse proxy mode
Assuming someone starts mitmproxy in reverse proxy mode with recorded flows, e.g.
`mitmproxy -r recorded_flows -R http://127.0.0.1:80`
How do we react to replays? Do we send them to the original server or do we send them to the reverse proxy?

Assuming someone starts mitmproxy in upstream proxy mode with recorded flows, e.g.
`mitmproxy -r recorded_flows -U http://127.0.0.1:8080`
How do we react to replays? Do we send them to the original server or do we send them to the upstream proxy?



Discussion:
I changed the implementation to the following:
**Reverse Proxy Mode:** We replay the request to the original URL, which is also displayed in the UI. While replaying to the reverse proxy server may be preferrable in some cases, this is pretty much an edge case and the former solution is/was easier to implement.
**Upstream Proxy Mode:** We replay the request through the upstream proxy.

@cortesi: Left open for your information, feel free to close (or dispute :stuck_out_tongue:).


Resolving patch:
diff --git a/libmproxy/flow.py b/libmproxy/flow.py
--- a/libmproxy/flow.py
+++ b/libmproxy/flow.py
@@ -615,6 +615,11 @@ def load_flow(self, f):
         """
             Loads a flow, and returns a new flow object.
         """
+
+        if self.server and self.server.config.mode == "reverse":
+            f.request.host, f.request.port = self.server.config.mode.dst[2:]
+            f.request.scheme = "https" if self.server.config.mode.dst[1] else "http"
+
         f.reply = controller.DummyReply()
         if f.request:
             self.handle_request(f)


Test patch:
diff --git a/test/test_flow.py b/test/test_flow.py
--- a/test/test_flow.py
+++ b/test/test_flow.py
@@ -5,8 +5,9 @@
 from libmproxy import filt, protocol, controller, utils, tnetstring, flow
 from libmproxy.protocol.primitives import Error, Flow
 from libmproxy.protocol.http import decoded, CONTENT_MISSING
-from libmproxy.proxy.connection import ClientConnection, ServerConnection
-from netlib import tcp
+from libmproxy.proxy import ProxyConfig
+from libmproxy.proxy.server import DummyServer
+from libmproxy.proxy.connection import ClientConnection
 import tutils
 
 
@@ -491,6 +492,14 @@ def test_load_flows(self):
         fm.load_flows(r)
         assert len(s._flow_list) == 6
 
+    def test_load_flows_reverse(self):
+        r = self._treader()
+        s = flow.State()
+        conf = ProxyConfig(mode="reverse", upstream_server=[True,True,"use-this-domain",80])
+        fm = flow.FlowMaster(DummyServer(conf), s)
+        fm.load_flows(r)
+        assert s._flow_list[0].request.host == "use-this-domain"
+
     def test_filter(self):
         sio = StringIO()
         fl = filt.parse("~c 200")
Repository: mitmproxy/mitmproxy

Issue:
mitmproxy -c throws an exception.
running ./mitmproxy  -c file
throws an exception 

```
Traceback (most recent call last):
  File "./mitmproxy", line 3, in <module>
    mitmproxy()
  File "/home/gato/trabajos/mitmproxy/libmproxy/main.py", line 97, in mitmproxy
    m = console.ConsoleMaster(server, console_options)
  File "/home/gato/trabajos/mitmproxy/libmproxy/console/__init__.py", line 191, in __init__
    self.client_playback_path(options.client_replay)
  File "/home/gato/trabajos/mitmproxy/libmproxy/console/__init__.py", line 301, in client_playback_path
    flows = self._readflows(path)
  File "/home/gato/trabajos/mitmproxy/libmproxy/console/__init__.py", line 296, in _readflows
    return flow.read_flows_from_paths([path])
  File "/home/gato/trabajos/mitmproxy/libmproxy/flow.py", line 998, in read_flows_from_paths
    path = os.path.expanduser(path)
  File "/usr/lib/python2.7/posixpath.py", line 261, in expanduser
    if not path.startswith('~'):
AttributeError: 'list' object has no attribute 'startswith'
```

running ./mitmdump  -c file works just fine.



Discussion:

Resolving patch:
diff --git a/libmproxy/console/__init__.py b/libmproxy/console/__init__.py
--- a/libmproxy/console/__init__.py
+++ b/libmproxy/console/__init__.py
@@ -293,16 +293,20 @@ def _readflows(self, path):
             - a list of flows, otherwise.
         """
         try:
-            return flow.read_flows_from_paths([path])
+            return flow.read_flows_from_paths(path)
         except flow.FlowReadError as e:
             signals.status_message.send(message=e.strerror)
 
     def client_playback_path(self, path):
+        if not isinstance(path, list):
+            path = [path]
         flows = self._readflows(path)
         if flows:
             self.start_client_playback(flows, False)
 
     def server_playback_path(self, path):
+        if not isinstance(path, list):
+            path = [path]
         flows = self._readflows(path)
         if flows:
             self.start_server_playback(


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmproxy console: client playback indicator should be cleared after playback completes
Start a client playback. Observe that the client playback indicator persists after playback completes. Pressing "c" again to stop client playback does not clear the indicator either.



Discussion:

Resolving patch:
diff --git a/libmproxy/flow.py b/libmproxy/flow.py
--- a/libmproxy/flow.py
+++ b/libmproxy/flow.py
@@ -771,6 +771,8 @@ def tick(self, q, timeout):
             if all(e):
                 self.shutdown()
             self.client_playback.tick(self)
+            if self.client_playback.done():
+                self.client_playback = None
 
         return super(FlowMaster, self).tick(q, timeout)
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Clicks on empty space can unintentionally open a new view
I often click on the terminal window to activate keyboard focus.

The new click feature in mitmproxy often causes my click to open an unwanted view.

This issue is especially problematic on the flow list screen where a click almost anywhere on the screen takes you to a flow view. Even clicks on empty space surrounding the flow causes navigation.

Is there a way to ignore clicks on empty spaces? Alternatively, can the mouse events be turned off?



Discussion:
The click zone is the entirety of the flow widget, which can extend beyond the visible text. I would merge a patch to add a command-line switch to turn off mouse support. It would be a very simple patch, and a great first contribution to the project.


Resolving patch:
diff --git a/libmproxy/cmdline.py b/libmproxy/cmdline.py
--- a/libmproxy/cmdline.py
+++ b/libmproxy/cmdline.py
@@ -608,6 +608,11 @@ def mitmproxy():
         action="store_true", dest="eventlog",
         help="Show event log."
     )
+    parser.add_argument(
+        "--no-mouse",
+        action="store_true", dest="no_mouse",
+        help="Disable mouse interaction."
+    )
     group = parser.add_argument_group(
         "Filters",
         "See help in mitmproxy for filter expression syntax."
diff --git a/libmproxy/console/__init__.py b/libmproxy/console/__init__.py
--- a/libmproxy/console/__init__.py
+++ b/libmproxy/console/__init__.py
@@ -154,7 +154,8 @@ class Options(object):
         "wfile",
         "nopop",
         "palette",
-        "palette_transparent"
+        "palette_transparent",
+        "no_mouse"
     ]
 
     def __init__(self, **kwargs):
@@ -455,12 +456,12 @@ def ticker(self, *userdata):
 
     def run(self):
         self.ui = urwid.raw_display.Screen()
-        self.ui.set_mouse_tracking()
         self.ui.set_terminal_properties(256)
         self.set_palette(self.palette)
         self.loop = urwid.MainLoop(
             urwid.SolidFill("x"),
             screen = self.ui,
+            handle_mouse = not self.options.no_mouse,
         )
 
         self.server.start_slave(
diff --git a/libmproxy/main.py b/libmproxy/main.py
--- a/libmproxy/main.py
+++ b/libmproxy/main.py
@@ -55,6 +55,7 @@ def mitmproxy(args=None):  # pragma: nocover
     console_options.eventlog = options.eventlog
     console_options.intercept = options.intercept
     console_options.limit = options.limit
+    console_options.no_mouse = options.no_mouse
 
     server = get_server(console_options.no_server, proxy_config)
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Can't parse unicode into character when viewing HTML content
### Step to reproduce the problem:
1. Open a HTML webpage in Chinese with mitmproxy.
2. Switch to the _Response_ page and view the content.
### What is the expected behavior?

Should display the content like 

``` html
<meta name="description" content="中文字符"/>
```

instead of

``` html
<meta name="description" content="&#20013;&#25991;&#23383;&#31526;"/>
```
### What went wrong?

I guess `libmproxy.contentviews.ViewHTML` should be improved to fix this. 
### Any other comments?

It's really confusing seeing whole screen of these unicodes instead of the correct characters. Hope we can fix this soon!

Can't parse unicode into character when viewing HTML content
### Step to reproduce the problem:
1. Open a HTML webpage in Chinese with mitmproxy.
2. Switch to the _Response_ page and view the content.
### What is the expected behavior?

Should display the content like 

``` html
<meta name="description" content="中文字符"/>
```

instead of

``` html
<meta name="description" content="&#20013;&#25991;&#23383;&#31526;"/>
```
### What went wrong?

I guess `libmproxy.contentviews.ViewHTML` should be improved to fix this. 
### Any other comments?

It's really confusing seeing whole screen of these unicodes instead of the correct characters. Hope we can fix this soon!



Discussion:

Resolving patch:
diff --git a/libmproxy/contentviews.py b/libmproxy/contentviews.py
--- a/libmproxy/contentviews.py
+++ b/libmproxy/contentviews.py
@@ -228,7 +228,8 @@ def __call__(self, data, **metadata):
             s = lxml.etree.tostring(
                 d,
                 pretty_print=True,
-                doctype=docinfo.doctype
+                doctype=docinfo.doctype,
+                encoding='utf8'
             )
             return "HTML", format_text(s)
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
ServerException instead of ProxyServerError
##### Steps to reproduce the problem:

```
>>> from libmproxy.proxy.server import ProxyServer
>>> from libmproxy.proxy.config import ProxyConfig
>>> ProxyServer(ProxyConfig(port=80))
(...)
ServerException: Error starting proxy server: error(13, 'Permission denied')
```
##### What is the expected behavior?

According to the documentation:

```
>>> ProxyServer?                       
Type:            type
String form:     <class 'libmproxy.proxy.server.ProxyServer'>
File:            /usr/lib/python2.7/dist-packages/libmproxy/proxy/server.py
Init definition: ProxyServer(self, config)
Docstring:       <no docstring>
Init docstring:  Raises ProxyServerError if there's a startup problem.
```

the expected behavior is 

```
>>> ProxyServer(ProxyConfig(port=80))
(...)
ProxyServerError: Error starting proxy server: error(13, 'Permission denied')
```
##### What went wrong?

Maybe the documentation is wrong?
##### Any other comments?

Nope.

---

Mitmproxy Version: 0.15-2
Operating System: Debian Sid.



Discussion:
Thanks. It looks like your mitmproxy is outdated. Could you please upgrade an try again?

I think the docstring is still there:
https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/proxy/server.py#L39

Thanks. Could you please send a quick PR that updates the docstring?


Resolving patch:
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -36,7 +36,7 @@ class ProxyServer(tcp.TCPServer):
 
     def __init__(self, config):
         """
-            Raises ProxyServerError if there's a startup problem.
+            Raises ServerException if there's a startup problem.
         """
         self.config = config
         try:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Headers aren't always case insensitive
Headers like `Content-Length` should preserve case after going trough mitmproxy, because some versions of PHP don't like lowercase headers (this is a server-side misimplementation of [RFC 2616](https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2), but it would be nice if mitmproxy was able to handle this.).  

Steps to reproduce:
Capture some traffic, replay it with `mitmdump -nc capture_file`. Everything works as intended.
Now replay it, but use a script that replaces the request content. The `Content-Length` header get's changed to `content-lenght`.  

The problem probably lies at [netlib/http/message.py](https://github.com/mitmproxy/mitmproxy/blob/master/netlib/http/message.py#L104)
Mitmproxy Version: latest



Discussion:
Does this mean it is not possible to use HTTP/2 with PHP?

From RFC750, 8.1.2:

> However, header field names MUST be converted to lowercase prior to their
> encoding in HTTP/2.  A request or response containing uppercase
> header field names MUST be treated as malformed (Section 8.1.2.6).

Probably not with all servers, but with this specific (HTTP/1), PHP server (I have no access to sourcecode), it only works if the header is "properly" capitalized. (I sent the exact same packets with and without capitalization, it only works with).

Just confirmed: changing `content-length` to `Content-Length` on https://github.com/mitmproxy/mitmproxy/blob/master/netlib/http/message.py#L104 fixed the issue. This is however probably not a solution since there is maybe a server which only accepts lowercase header. A good solution would be to preserve case.

If you replace `key` in [this line](https://github.com/mitmproxy/mitmproxy/blob/ef462a05d7422ec099cafdd52810fbb77dbad604/netlib/multidict.py#L108) with `fields[0]`, does that fix your issue?
If so, It'd be awesome if you could send a PR for that! :smiley:


Resolving patch:
diff --git a/netlib/multidict.py b/netlib/multidict.py
--- a/netlib/multidict.py
+++ b/netlib/multidict.py
@@ -105,7 +105,7 @@ def set_all(self, key, values):
             if self._kconv(field[0]) == key_kconv:
                 if values:
                     new_fields.append(
-                        (key, values.pop(0))
+                        (field[0], values.pop(0))
                     )
             else:
                 new_fields.append(field)


Test patch:
diff --git a/test/netlib/test_multidict.py b/test/netlib/test_multidict.py
--- a/test/netlib/test_multidict.py
+++ b/test/netlib/test_multidict.py
@@ -120,7 +120,7 @@ def test_set_all(self):
             ("a", "b"),
             ("x", "x"),
             ("c", "d"),
-            ("X", "x"),
+            ("X", "X"),
             ("e", "f"),
         ))
         md.set_all("x", ["1", "2", "3"])
@@ -128,7 +128,7 @@ def test_set_all(self):
             ("a", "b"),
             ("x", "1"),
             ("c", "d"),
-            ("x", "2"),
+            ("X", "2"),
             ("e", "f"),
             ("x", "3"),
         )