internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: electricitymaps/electricitymaps-contrib

Issue:
KR production parser down
## Description

This is an automatic error report generated for South Korea (KR).

Issues:
- No recent data found for `consumption` parser
- No recent data found for `price` parser
- No recent data found for `production` parser

## Suggestions
- Try running the parser locally using the command `poetry run test_parser KR production`
- <a href="https://storage.googleapis.com/electricitymap-parser-logs/KR.html">Explore the runtime logs</a>

You can see an overview of all parser issues [here](https://github.com/tmrowco/electricitymap-contrib/wiki/Parser-issues).



Discussion:

Resolving patch:
diff --git a/parsers/KR.py b/parsers/KR.py
--- a/parsers/KR.py
+++ b/parsers/KR.py
@@ -90,7 +90,7 @@ def fetch_consumption(
     r = session or Session()
     url = REAL_TIME_URL
 
-    response = r.get(url)
+    response = r.get(url, verify=False)
     assert response.status_code == 200
 
     soup = BeautifulSoup(response.text, "html.parser")
@@ -139,7 +139,7 @@ def fetch_price(
     r = session or Session()
     url = PRICE_URL
 
-    response = r.get(url)
+    response = r.get(url, verify=False)
     assert response.status_code == 200
 
     all_data = []
@@ -246,7 +246,7 @@ def get_long_term_prod_data(
 
 def get_granular_real_time_prod_data(session: Optional[Session] = None) -> dict:
     r0 = session or Session()
-    res_0 = r0.get(REAL_TIME_URL)
+    res_0 = r0.get(REAL_TIME_URL, verify=False)
     chart_data = extract_chart_data(res_0.text)
 
     return chart_data
@@ -285,7 +285,7 @@ def fetch_production(
                 "source": "https://new.kpx.or.kr",
             }
 
-            data["storage"]["hydro"] = chart_data_values["pumpedHydro"]
+            data["storage"]["hydro"] = -chart_data_values["pumpedHydro"]
 
             data["production"]["coal"] = chart_data_values["coal"]
             data["production"]["gas"] = chart_data_values["gas"]


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Wrong data in Spain<->Morocco connection
## Description
Just as I suspected, the data for the Spain<->Morocco connection is wrong, as can be seen on the [ESIOS webpage](https://www.esios.ree.es/en/analysis/10209) (ignore the lack of data during last night, that's not linked to this).
It is related to the fact that the parser returns MWh of fifteen minutes intervals, while the backend expects MW, so averages instead of adding.
Apart from changing something in the backend, the easiest solution that I found is forcing the API to return hour intervals. We would have to add a small delay also, to prevent returning incomplete data. (e.g. server only adds 11:00 and 11:15 if the provided date is 11:24)
We could alternatively multiply the 15 minutes interval data by 4 to get the MW. This would increase the granularity but would imply adding specific cases for this exchange, the parser would break if the granularity ever changes, etc. Whereas by forcing the granularity to 1 hour we make sure the granularity is always correct.
What is your opinion?


Discussion:
I think we would have to use 15 minutes values going forward otherwise only every 4th value would be overwritten in the database when I refetch all the data. 

So adding a function to multiply it by 4 is probably the best option. I think adding a table similar to the one we did for the ID but with multiplication factors. That way we could support hourly down to per minute if needed.

I'll try and take a look at it later today as well. 
I can work on a PR to add the table if you will.
> I can work on a PR to add the table if you will.

That would be great! 🎉 

Resolving patch:
diff --git a/parsers/ESIOS.py b/parsers/ESIOS.py
--- a/parsers/ESIOS.py
+++ b/parsers/ESIOS.py
@@ -24,6 +24,13 @@
     "ES->MA": "10209",
 }
 
+# Map each exchange to the needed factor to adjust from MWh to MW. Depends on the time granularity of the API for each request
+# E.g ES->MA is 4 because the API returns 15 minutes intervals data (15 min = 1/4 of an hour; P=E/t).
+EXCHANGE_MULTIPLICATION_FACTOR_MAP = {
+    "AD->ES": 1,
+    "ES->MA": 4,
+}
+
 
 def format_url(target_datetime: datetime, ID: str):
     start_date = (target_datetime - timedelta(hours=24)).isoformat()
@@ -55,7 +62,10 @@ def fetch_exchange(
     }
 
     zone_key = ZoneKey("->".join(sorted([zone_key1, zone_key2])))
-    if zone_key not in EXCHANGE_ID_MAP.keys():
+    if (
+        zone_key not in EXCHANGE_ID_MAP
+        or zone_key not in EXCHANGE_MULTIPLICATION_FACTOR_MAP
+    ):
         raise ParserException(
             "ESIOS.py",
             f"This parser cannot parse data between {zone_key1} and {zone_key2}.",
@@ -82,6 +92,8 @@ def fetch_exchange(
             -value["value"] if zone_key.partition("->")[0] == "ES" else value["value"]
         )
 
+        net_flow *= EXCHANGE_MULTIPLICATION_FACTOR_MAP[zone_key]
+
         exchanges.append(
             zoneKey=zone_key,
             datetime=arrow.get(value["datetime_utc"]).datetime,


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Data for French Guyana (up to today)
Hourly production data for French Guyana: https://opendata-guyane.edf.fr/explore/dataset/production-d-electricite-par-filiere-en-temps-reel/

This one is up-to-date to today!

Also mentioned in #1912, but that was closed due to inactivity after some of the data sources were implemented. Also, the URLs have changed.

Edit: looks like the FR_O parser supports a 'historical dataset', but not this real-time dataset.


Discussion:
Thanks for flagging the updated data set! Should be pretty straight forward to add to the parser!

Resolving patch:
diff --git a/parsers/FR_O.py b/parsers/FR_O.py
--- a/parsers/FR_O.py
+++ b/parsers/FR_O.py
@@ -29,6 +29,8 @@
     "FR-COR": "production-delectricite-par-filiere-en-temps-reel",
     "GP": "mix-temps-reel-guadeloupe",
     "RE": "prod-electricite-temps-reel",
+    "GF": "production-d-electricite-par-filiere-en-temps-reel",
+    "MQ": "production-delectricite-par-filiere-en-temps-reel",
 }
 
 HISTORICAL_DATASETS = {
@@ -68,7 +70,7 @@
             "micro_hydro",
             "micro_hydraulique_mw",
         ],
-        "oil": ["diesel", "moteur_diesel"],
+        "oil": ["diesel", "moteur_diesel", "centrale_au_fioul", "moteurs_diesels"],
         "solar": [
             "photovoltaique",
             "photovoltaique0",
@@ -133,7 +135,7 @@ def fetch_data(
         "FR-COR": "date_heure" if target_datetime else "date",
         "RE": "date_heure" if target_datetime else "date",
         "GF": "date",
-        "MQ": "date_heure",
+        "MQ": "date_heure" if target_datetime else "date",
         "GP": "date",
     }
 
@@ -229,7 +231,7 @@ def fetch_production(
             storage=storage,
             source=generate_source(zone_key),
             sourceType=EventSourceType.estimated
-            if production_object["statut"] == "Estimé"
+            if production_object.get("statut") == "Estimé"
             else EventSourceType.measured,
         )
     return production_breakdown_list.to_list()


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
PH-VI production parser down
## Description

This is an automatic error report generated for Philippines Visayas (PH-VI).

Issues:
- No recent data found for `production` parser

## Suggestions
- Try running the parser locally using the command `poetry run test_parser PH-VI production`
- <a href="https://storage.googleapis.com/electricitymap-parser-logs/PH-VI.html">Explore the runtime logs</a>

You can see an overview of all parser issues [here](https://github.com/electricitymaps/electricitymaps-contrib/wiki/Parser-issues).



Discussion:
@mathilde-daugy FYI the update with battery must not have been tested thoroughly.
I've got a fix ready, but would you have time today to add a snapshot test on `fetch_production`? 
Can do it together with you

Resolving patch:
diff --git a/parsers/PH.py b/parsers/PH.py
--- a/parsers/PH.py
+++ b/parsers/PH.py
@@ -617,9 +617,13 @@ def pivot_per_mode(df: pd.DataFrame) -> pd.DataFrame:
     # Flatten columns and make them "production.{mode}"
     df.columns = df.columns.to_series().str.join(".")
     # Handle storage
-    if "production.hydro_storage" in df.columns:
-        df = df.rename(columns={"production.hydro_storage": "storage.hydro"})
-        df["storage.hydro"] *= -1
+    storage_methods = {"hydro_storage": "hydro", "battery": "battery"}
+    for storage_method, storage_mode in storage_methods.items():
+        if f"production.{storage_method}" in df.columns:
+            df = df.rename(
+                columns={f"production.{storage_method}": f"storage.{storage_mode}"}
+            )
+            df[f"storage.{storage_mode}"] *= -1
     return df
 
 
@@ -690,8 +694,8 @@ def fetch_production(
         for mode in [m for m in row.index if "production." in m]:
             production_mix.add_value(mode.replace("production.", ""), row[mode])
         storage_mix = StorageMix()
-        if "storage.hydro" in row.index:
-            storage_mix.add_value("hydro", row["storage.hydro"])
+        for mode in [m for m in row.index if "storage." in m]:
+            storage_mix.add_value(mode.replace("storage.", ""), row[mode])
         production_breakdown.append(
             zone_key,
             tstamp.to_pydatetime(),


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
GB production parser down
## Description

This is an automatic error report generated for Great Britain (GB).

Issues:
- No recent data found for `consumption` parser
- No recent data found for `consumptionForecast` parser
- No recent data found for `generationForecast` parser
- No recent data found for `price` parser
- No recent data found for `production` parser
- No recent data found for `productionPerModeForecast` parser

## Suggestions
- Try running the parser locally using the command `poetry run test_parser GB production`
- <a href="https://storage.googleapis.com/electricitymap-parser-logs/GB.html">Explore the runtime logs</a>

You can see an overview of all parser issues [here](https://github.com/electricitymaps/electricitymaps-contrib/wiki/Parser-issues).



Discussion:

Resolving patch:
diff --git a/parsers/ELEXON.py b/parsers/ELEXON.py
--- a/parsers/ELEXON.py
+++ b/parsers/ELEXON.py
@@ -37,8 +37,8 @@
 
 REPORT_META = {
     "B1620": {"expected_fields": 13, "skiprows": 5},
-    "FUELINST": {"expected_fields": 22, "skiprows": 1},
-    "INTERFUELHH": {"expected_fields": 11, "skiprows": 0},
+    "FUELINST": {"expected_fields": 23, "skiprows": 1},
+    "INTERFUELHH": {"expected_fields": 12, "skiprows": 0},
 }
 
 # 'hydro' key is for hydro production
@@ -76,6 +76,7 @@
     "INTELEC": "exchange",
     "INTIFA2": "exchange",
     "INTNSL": "exchange",
+    "INTVKL": "exchange",
 }
 
 ESO_FUEL_MAPPING = {
@@ -91,6 +92,7 @@
     "GB->IE": [6],
     "BE->GB": [7],
     "GB->NO-NO2": [10],  # North Sea Link
+    "DK-DK1->GB": [11],  # Viking Link
 }
 
 
@@ -544,3 +546,6 @@ def fetch_production(
 
     print("fetch_exchange(GB, NL) ->")
     print(fetch_exchange("GB", "NL"))
+
+    print("fetch_exchange(GB, DK-DK1) ->")
+    print(fetch_exchange("DK-DK1", "GB"))


Test patch:
Repository: tornadoweb/tornado

Issue:
Low perfomance on big chunks writing
_merge_buffer produces too much memcpy operations on heavy data chunks if you pass them to IOStream.write

Details: https://groups.google.com/d/topic/python-tornado/NEgKGiFkHXI/discussion



Discussion:

Resolving patch:
diff --git a/tornado/iostream.py b/tornado/iostream.py
--- a/tornado/iostream.py
+++ b/tornado/iostream.py
@@ -192,7 +192,11 @@ def write(self, data, callback=None):
         if data:
             # We use bool(_write_buffer) as a proxy for write_buffer_size>0,
             # so never put empty strings in the buffer.
-            self._write_buffer.append(data)
+            if len(data) > 128*1024:
+                for i in range(0, len(data), 128*1024):
+                    self._write_buffer.append(data[i:i+128*1024])
+            else:
+                self._write_buffer.append(data)
         self._write_callback = stack_context.wrap(callback)
         self._handle_write()
         if self._write_buffer:


Test patch:
Repository: tornadoweb/tornado

Issue:
Windows: SO_REUSEADDR allows listening on occupied port
Context:

Try to run two instances of helloworld.py example

Symptom:

Linux/OS X (expected): second instance raises `Address already in use`.
Windows: second instance runs happily, but receives no requests until the first instance shuts down.

Cause:

[Setting SO_REUSEADDR](https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L255) seems to result in different behavior between Windows and other platforms.

I don't know if it makes sense to simply skip setting REUSEADDR on Windows or what, but it would be nice if tornado's behavior was consistent.



Discussion:
What about TIME_WAIT?  If you don't use SO_REUSEADDR do you sometimes have to wait for the previous incarnation's bindings to expire before you can restart a server?  That seems worse than allowing a second non-functioning process to start up. 

Windows is broken - it interprets SO_REUSEADDR differently from POSIX. From reading [this Python bug](http://bugs.python.org/issue2550) and [the Microsoft documentation](http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621.aspx), I believe the fix is to set SO_EXCLUSIVEADDRUSE instead on Windows (the only platform it exists on), so that EADDRINUSE is raised.


Resolving patch:
diff --git a/tornado/netutil.py b/tornado/netutil.py
--- a/tornado/netutil.py
+++ b/tornado/netutil.py
@@ -252,7 +252,8 @@ def bind_sockets(port, address=None, family=socket.AF_UNSPEC, backlog=128):
         af, socktype, proto, canonname, sockaddr = res
         sock = socket.socket(af, socktype, proto)
         set_close_exec(sock.fileno())
-        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        if os.name != 'nt':
+            sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         if af == socket.AF_INET6:
             # On linux, ipv6 sockets accept ipv4 too by default,
             # but this makes it impossible to bind to both


Test patch:
Repository: tornadoweb/tornado

Issue:
remote_ip breaks if "X-Forwarded-For" is a list
From the mailing list:

The "X-Forwarded-For" header can be a comma-separated list of IP addresses. This happens when Tornado is behind a reverse proxy such as HAProxy.  The first IP in the list will be the original client IP, and the last one will be the proxy closest to Tornado.

After this change, this comma-separated list is not considered valid, and so it's discarded. Then remote_ip falls back to the IP of the proxy, which is not what we want.

As a result we have to write our own method to return the IP and use that everywhere. It would be nice if Tornado could handle it, especially to make it easier for others so they don't have to figure out this problem themselves.



Discussion:

Resolving patch:
diff --git a/tornado/httpserver.py b/tornado/httpserver.py
--- a/tornado/httpserver.py
+++ b/tornado/httpserver.py
@@ -431,8 +431,10 @@ def __init__(self, method, uri, version="HTTP/1.0", headers=None,
         # xheaders can override the defaults
         if connection and connection.xheaders:
             # Squid uses X-Forwarded-For, others use X-Real-Ip
+            ip = self.headers.get("X-Forwarded-For", self.remote_ip)
+            ip = ip.split(',')[-1].strip()
             ip = self.headers.get(
-                "X-Real-Ip", self.headers.get("X-Forwarded-For", self.remote_ip))
+                "X-Real-Ip", ip)
             if netutil.is_valid_ip(ip):
                 self.remote_ip = ip
             # AWS uses X-Forwarded-Proto


Test patch:
diff --git a/tornado/test/httpserver_test.py b/tornado/test/httpserver_test.py
--- a/tornado/test/httpserver_test.py
+++ b/tornado/test/httpserver_test.py
@@ -397,16 +397,31 @@ def test_ip_headers(self):
             self.fetch_json("/", headers=valid_ipv4)["remote_ip"],
             "4.4.4.4")
 
+        valid_ipv4_list = {"X-Forwarded-For": "127.0.0.1, 4.4.4.4"}
+        self.assertEqual(
+            self.fetch_json("/", headers=valid_ipv4_list)["remote_ip"],
+            "4.4.4.4")
+
         valid_ipv6 = {"X-Real-IP": "2620:0:1cfe:face:b00c::3"}
         self.assertEqual(
             self.fetch_json("/", headers=valid_ipv6)["remote_ip"],
             "2620:0:1cfe:face:b00c::3")
 
+        valid_ipv6_list = {"X-Forwarded-For": "::1, 2620:0:1cfe:face:b00c::3"}
+        self.assertEqual(
+            self.fetch_json("/", headers=valid_ipv6_list)["remote_ip"],
+            "2620:0:1cfe:face:b00c::3")
+
         invalid_chars = {"X-Real-IP": "4.4.4.4<script>"}
         self.assertEqual(
             self.fetch_json("/", headers=invalid_chars)["remote_ip"],
             "127.0.0.1")
 
+        invalid_chars_list = {"X-Forwarded-For": "4.4.4.4, 5.5.5.5<script>"}
+        self.assertEqual(
+            self.fetch_json("/", headers=invalid_chars_list)["remote_ip"],
+            "127.0.0.1")
+
         invalid_host = {"X-Real-IP": "www.google.com"}
         self.assertEqual(
             self.fetch_json("/", headers=invalid_host)["remote_ip"],
Repository: tornadoweb/tornado

Issue:
Typo on line tornado/iostream.py:334
Typo on line tornado/iostream.py:334

```
    if data:
        if (self.max_write_buffer_size is not None and
                self._write_buffer_size + len(data) > self.max_write_buffer_size):
            raise StreamBufferFullError("Reached maximum read buffer size")
```

The Error message of `StreamBufferFullError` should be "Reached maximum write buffer size".



Discussion:

Resolving patch:
diff --git a/tornado/iostream.py b/tornado/iostream.py
--- a/tornado/iostream.py
+++ b/tornado/iostream.py
@@ -331,7 +331,7 @@ def write(self, data, callback=None):
         if data:
             if (self.max_write_buffer_size is not None and
                     self._write_buffer_size + len(data) > self.max_write_buffer_size):
-                raise StreamBufferFullError("Reached maximum read buffer size")
+                raise StreamBufferFullError("Reached maximum write buffer size")
             # Break up large contiguous strings before inserting them in the
             # write buffer, so we don't have to recopy the entire thing
             # as we slice off pieces to send to the socket.


Test patch:
Repository: tornadoweb/tornado

Issue:
redirect requests starting with '//' to '/' leading to wrong place
Tornado uses `urljoin` to join `self.request.uri` and the destination but when `self.request.uri` starts with '//' it generates locations still start with '//' because this behaviour of `urljoin`:

```
>>> from urllib.parse import urljoin                                                                                                                   
>>> urljoin('//abc', '/abc')
'//abc/abc'
```

I suggest using `self.request.full_url()` instead. Also, the HTTP specification says that the Location header should include the host part.

PS: `self.request.full_url()` doesn't work for proxy requests that have full urls in their request line.



Discussion:

Resolving patch:
diff --git a/tornado/web.py b/tornado/web.py
--- a/tornado/web.py
+++ b/tornado/web.py
@@ -650,8 +650,7 @@ def redirect(self, url, permanent=False, status=None):
         else:
             assert isinstance(status, int) and 300 <= status <= 399
         self.set_status(status)
-        self.set_header("Location", urlparse.urljoin(utf8(self.request.uri),
-                                                     utf8(url)))
+        self.set_header("Location", utf8(url))
         self.finish()
 
     def write(self, chunk):


Test patch:
diff --git a/tornado/test/web_test.py b/tornado/test/web_test.py
--- a/tornado/test/web_test.py
+++ b/tornado/test/web_test.py
@@ -597,6 +597,7 @@ def get_handlers(self):
             url("/redirect", RedirectHandler),
             url("/web_redirect_permanent", WebRedirectHandler, {"url": "/web_redirect_newpath"}),
             url("/web_redirect", WebRedirectHandler, {"url": "/web_redirect_newpath", "permanent": False}),
+            url("//web_redirect_double_slash", WebRedirectHandler, {"url": '/web_redirect_newpath'}),
             url("/header_injection", HeaderInjectionHandler),
             url("/get_argument", GetArgumentHandler),
             url("/get_arguments", GetArgumentsHandler),
@@ -730,6 +731,11 @@ def test_web_redirect(self):
         self.assertEqual(response.code, 302)
         self.assertEqual(response.headers['Location'], '/web_redirect_newpath')
 
+    def test_web_redirect_double_slash(self):
+        response = self.fetch("//web_redirect_double_slash", follow_redirects=False)
+        self.assertEqual(response.code, 301)
+        self.assertEqual(response.headers['Location'], '/web_redirect_newpath')
+
     def test_header_injection(self):
         response = self.fetch("/header_injection")
         self.assertEqual(response.body, b"ok")
Repository: tornadoweb/tornado

Issue:
Options should support setitem
From http://stackoverflow.com/questions/33411269/attributeerror-in-python-tornado-to-configure-log-into-a-file/33417289

Options currently override getattr, setattr, and getitem. Item and attribute syntax should be symmetric so we need a setitem override to match setattr.



Discussion:
I could handle this, would you like me to?

Feel free, although I was actually thinking of tagging this one as a good entry point for new contributors. 

OK, I'll leave it unresolved then. =)


Resolving patch:
diff --git a/tornado/options.py b/tornado/options.py
--- a/tornado/options.py
+++ b/tornado/options.py
@@ -132,8 +132,10 @@ def __contains__(self, name):
         return name in self._options
 
     def __getitem__(self, name):
-        name = self._normalize_name(name)
-        return self._options[name].value()
+        return self.__getattr__(name)
+
+    def __setitem__(self, name, value):
+        return self.__setattr__(name, value)
 
     def items(self):
         """A sequence of (name, value) pairs.


Test patch:
diff --git a/tornado/test/options_test.py b/tornado/test/options_test.py
--- a/tornado/test/options_test.py
+++ b/tornado/test/options_test.py
@@ -131,6 +131,12 @@ def test_getitem(self):
         options = self._sample_options()
         self.assertEqual(1, options['a'])
 
+    def test_setitem(self):
+        options = OptionParser()
+        options.define('foo', default=1, type=int)
+        options['foo'] = 2
+        self.assertEqual(options['foo'], 2)
+
     def test_items(self):
         options = self._sample_options()
         # OptionParsers always define 'help'.
Repository: tornadoweb/tornado

Issue:
current_user's docstring example has unexpected indent
In [web.py L1075](https://github.com/tornadoweb/tornado/blob/master/tornado/web.py#L1075) the example code has unexpected indent.



Discussion:

Resolving patch:
diff --git a/tornado/web.py b/tornado/web.py
--- a/tornado/web.py
+++ b/tornado/web.py
@@ -1072,8 +1072,8 @@ def current_user(self):
 
               def get_current_user(self):
                   user_cookie = self.get_secure_cookie("user")
-                      if user_cookie:
-                          return json.loads(user_cookie)
+                  if user_cookie:
+                      return json.loads(user_cookie)
                   return None
 
         * It may be set as a normal variable, typically from an overridden


Test patch: