internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Costa Rica does not have data available between 12am-1am
```
Traceback (most recent call last):
  File "/home/feeder/lib/fetch_data.py", line 130, in launch_parsers
    **parser_kwargs)
  File "/home/electricitymap/parsers/CR.py", line 191, in fetch_production
    df = pd.read_html(response.text, match=CHARACTERISTIC_NAME, skiprows=1, index_col=0)[0]
  File "/usr/local/lib/python3.6/site-packages/pandas/io/html.py", line 915, in read_html
    keep_default_na=keep_default_na)
  File "/usr/local/lib/python3.6/site-packages/pandas/io/html.py", line 749, in _parse
    raise_with_traceback(retained)
  File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 385, in raise_with_traceback
    raise exc.with_traceback(traceback)
ValueError: No tables found matching pattern 'Angostura'
```


Discussion:
This has happened every day between 7am and 8am for the past 90 days. There is no data table available in that timespan.  Costa Rica's timezone is UTC-6 and with daylight savings in Europe atm it correlates perfectly with no data being available for 12-1am in Costa Rica.

Possible solutions;

- If current time being requested is between 12-1am local time then get data for the previous day.
- Ignore it since the data backfills properly later on anyway. (assuming I'm right @corradio ?)
You're right assuming the CR parser returns a list of datapoints (in opposition to a single one)
Yes it returns a list of them.

Resolving patch:
diff --git a/parsers/CR.py b/parsers/CR.py
--- a/parsers/CR.py
+++ b/parsers/CR.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 # coding=utf-8
 
+import datetime as dt
 import logging
 
 import arrow
@@ -160,9 +161,17 @@ def df_to_data(zone_key, day, df, logger):
 
 def fetch_production(zone_key='CR', session=None,
                      target_datetime=None, logger=logging.getLogger(__name__)):
-    # ensure we have an arrow object. if no target_datetime is specified, this defaults to now.
+    # ensure we have an arrow object.
+    # if no target_datetime is specified, this defaults to now.
     target_datetime = arrow.get(target_datetime).to(TIMEZONE)
 
+    # if before 01:30am on the current day then fetch previous day due to
+    # data lag.
+    today = arrow.get().to(TIMEZONE).date()
+    if target_datetime.date() == today:
+        target_datetime = target_datetime if target_datetime.time() >= dt.time(1, 30) \
+            else target_datetime.shift(days=-1)
+
     if target_datetime < arrow.get('2012-07-01'):
         # data availability limit found by manual trial and error
         logger.error('CR API does not provide data before 2012-07-01, '


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Import intensity could fallback on yearly averages when missing/unknown
When a country, or area, is importing electricity from another country and the exporting country's production sources are unknown, it seems as if the intensity of the imported electricity is set to be equal to the intensity of the importing country. But this is hardly meaningful. Would it be possible to set the unknown intensity of imported electricity to an average or mean value from a historical period? E.g. the last month or the same month last year. Or to the last available dataset (depending on how old that is).

I can see that it happens quite often for Norway, that "Data [is] temporarily unavailable". The intensity of the electricity exported to Sweden is low, while it is medium high when exported to West Denmark.


Discussion:
That's a very good idea, which I also wanted to implement. I think it should be quite easy to do on our side. I'll add it on our backlog so we discuss it internally.
@brunolajoie any take on this?
I think it's a very good idea too! 
It requires to compute a fixed default GHG intensity for all zone of electricityMap, I guess, right?
that's right
You forecast the co2 intensity for all areas. Couldn't you just use that when actual values are missing?
Indeed @eimose but it poses certain threats as people might scrape the results and not pay for our API :(
That makes sense. However, I would think the forecast is only valuable if it is for the following 48 hours and not if it is only for the current hour. In other words, you could make the carbon intensity forecast for the current hour available instead of the full forecast and then use this value for a country, if the data source fails for a period. This would also challenge you on speed, as you would need to make it for all countries. 

Resolving patch:
diff --git a/utils/config.py b/utils/config.py
--- a/utils/config.py
+++ b/utils/config.py
@@ -40,3 +40,22 @@ def relative_path(script_reference_path, rel_path):
 # we want neighbors to always be in the same order
 for zone, neighbors in ZONE_NEIGHBOURS.items():
     ZONE_NEIGHBOURS[zone] = sorted(neighbors)
+
+CO2EQ_PARAMETERS = json.load(open(relative_path(
+    __file__, '../config/co2eq_parameters.json')))
+
+def emission_factors(zone_key):
+    fallback_carbon_intensity = CO2EQ_PARAMETERS['fallbackZoneMixes'].get(zone_key, {}).get('carbonIntensity');
+    override = CO2EQ_PARAMETERS['emissionFactors']['zoneOverrides'].get(zone_key, {})
+    defaults = CO2EQ_PARAMETERS['emissionFactors']['defaults']
+    merged = {**defaults, **override}
+    if fallback_carbon_intensity:
+        merged['battery storage'] = {
+            'value': fallback_carbon_intensity,
+            'source': 'Annual carbon intensity'
+        }
+        merged['hydro storage'] = {
+            'value': fallback_carbon_intensity,
+            'source': 'Annual carbon intensity'
+        }
+    return dict([(k, (v or {}).get('value')) for (k, v) in merged.items()])


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Include Solar data for Dominican Republic
Two of the largest PV plants of the Carribean, "Montecristi" with 58 MW (next year up to 119 MW) and "Monte Plata" with 69 MW are located in the Dominican Republic.
https://www.pv-magazine.com/2018/07/25/fs-solar-commissions-caribbeans-largest-pv-plant/
http://www.gesyw.com/portfolio/project/56d9431a353dc

I had a look into our source, data is reported for these two large scale solar plants under the "termico" section.
![image](https://user-images.githubusercontent.com/25743609/50443181-6c1dfd00-0902-11e9-9b7c-f41b926033d1.png)

We should definitely add these two to the map!
I'll try to submit a PR ;)


Discussion:

Resolving patch:
diff --git a/parsers/DO.py b/parsers/DO.py
--- a/parsers/DO.py
+++ b/parsers/DO.py
@@ -51,6 +51,7 @@
                  u'ESTRELLA DEL MAR 2 CGN': 'gas',
                  u'ESTRELLA DEL MAR 2 SFO': 'oil',
                  u'ESTRELLA DEL MAR 2 SGN': 'gas',
+                 u'GENERACI\xD3N DE EMERGENCIA AES ANDR\xC9S': 'gas',
                  u'HAINA TG': 'oil',
                  u'INCA KM22': 'oil',
                  u'ITABO 1': 'coal',
@@ -332,7 +333,7 @@ def fetch_production(zone_key='DO', session=None, target_datetime=None, logger=l
               'hydro': hour.get('hydro', 0.0),
               'nuclear': 0.0,
               'oil': hour.get('oil', 0.0),
-              'solar': None,
+              'solar': hour.get('solar', 0.0),
               'wind': hour.get('wind', 0.0),
               'geothermal': 0.0,
               'unknown': hour.get('unknown', 0.0)


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Emissions from oil shale
Estonia produces electricity from oil shale.

This is a solid fossil fuel that is in the same statistical group with coal.

It would be correct to dislpay the emissions from oil shale under "coal" (solid) not "oil" (liquid).

Source: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:Fossil_fuel&oldid=120658


Discussion:
Thanks for notifying. Your source seems uncontroversial, I'll make the change!

Resolving patch:
diff --git a/parsers/ENTSOE.py b/parsers/ENTSOE.py
--- a/parsers/ENTSOE.py
+++ b/parsers/ENTSOE.py
@@ -51,13 +51,13 @@
 ENTSOE_PARAMETER_BY_DESC = {v: k for k, v in ENTSOE_PARAMETER_DESC.items()}
 ENTSOE_PARAMETER_GROUPS = {
     'biomass': ['B01', 'B17'],
-    'coal': ['B02', 'B05', 'B08'],
+    'coal': ['B02', 'B05', 'B07', 'B08'],
     'gas': ['B03', 'B04'],
     'geothermal': ['B09'],
     'hydro': ['B11', 'B12'],
     'hydro storage': ['B10'],
     'nuclear': ['B14'],
-    'oil': ['B06', 'B07'],
+    'oil': ['B06'],
     'solar': ['B16'],
     'wind': ['B18', 'B19'],
     'other': ['B20', 'B13', 'B15']


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Midnight glitch in UK-GB data
Reported wind generation drops to zero every day at midnight. 

<img width="361" alt="Screenshot 2019-05-04 at 13 22 17" src="https://user-images.githubusercontent.com/13368547/57179023-fd811b00-6e70-11e9-99f7-814c0cfcd3cd.png">

Possibly related to #1779?


Discussion:
And maybe also related to #1754
@lorrieq any ideas here?
Apologies for the late reply, been a bit out of the loop. I presume the issue is the lookback to yesterday needs to be more liberal or we change it to look back a set number of hours. Hopefully I will be able to take a look later.

![image](https://user-images.githubusercontent.com/6525086/58087877-67136000-7bb9-11e9-8b8d-2f97ba3ae9a3.png)


Resolving patch:
diff --git a/parsers/ELEXON.py b/parsers/ELEXON.py
--- a/parsers/ELEXON.py
+++ b/parsers/ELEXON.py
@@ -230,8 +230,7 @@ def _fetch_wind(target_datetime=None):
 
     # line up with B1620 (main production report) search range
     d = target_datetime.date()
-    offset = 1 if target_datetime.time() <= dt.time(0, 30) else 0
-    start = dt.datetime.combine(d - dt.timedelta(days=offset), dt.time(0))
+    start = d - dt.timedelta(hours=24)
     end = dt.datetime.combine(d + dt.timedelta(days=1), dt.time(0))
 
     session = requests.session()


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Data for Kuwait
Hi all,

I had found the real-time "system load" data for Kuwait a whole while ago, and I am about to add a parser for it.

There is no breakdown of the fuels, which doesn't add anything to the map, I am aware of this.
But I always think it's interesting just to look at the general behaviour of electricity systems, as this may contribute to evaluating energy transition strategies for certain countries or geographical regions.

For the carbon intensity, I'll use IEA's electricity data for 2017, which says 34.4% natural gas and 65.6% oil. No imports or exports to any neighbours, so it's all domestic generation, and we can just use 
`load = production`.

I'll submit a PR soon!


Discussion:

Resolving patch:
diff --git a/parsers/KW.py b/parsers/KW.py
new file mode 100644
--- /dev/null
+++ b/parsers/KW.py
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+# coding=utf-8
+
+# This parser returns Kuwait's electricity system load (assumed to be equal to electricity production)
+# Source: Ministry of Electricity and Water / State of Kuwait
+# URL: https://www.mew.gov.kw/en/
+# Scroll down to see the system load gauge
+# Shares of Electricity production in 2017: 65.6% oil, 34.4% gas (source: IEA; https://www.iea.org/statistics/?country=KUWAIT&indicator=ElecGenByFuel)
+
+import arrow
+import requests
+import re
+
+def fetch_production(zone_key='KW', session=None, logger=None):
+    r = session or requests.session()
+    url = 'https://www.mew.gov.kw/en'
+    response = r.get(url)
+    load = re.findall(r"\((\d{4,5})\)", response.text)
+    load = int(load[0])
+    production = {}
+    production['unknown'] = load
+    
+    datapoint = {
+        'zoneKey': zone_key,
+        'datetime': arrow.now('Asia/Kuwait').datetime,
+        'production': production,
+        'source': 'mew.gov.kw'
+    }
+
+    return datapoint
+
+if __name__ == '__main__':
+    """Main method, never used by the electricityMap backend, but handy for testing."""
+    
+    print('fetch_production() ->')
+    print(fetch_production())


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Nova Scotia new URL (CA_NS)
Quick note, I haven't had a chance to dig into it in detail yet:

Looks like the website has been redesigned and things shuffled around a bit.

New page URL is https://www.nspower.ca/clean-energy/todays-energy-stats

JSON URL is now https://www.nspower.ca/library/CurrentLoad/CurrentMix.json

JSON format looks similar to before so it might just be a matter of swapping the URL.


Discussion:

Resolving patch:
diff --git a/parsers/CA_NS.py b/parsers/CA_NS.py
--- a/parsers/CA_NS.py
+++ b/parsers/CA_NS.py
@@ -35,14 +35,11 @@ def _get_ns_info(requests_obj, logger):
         'wind': 700
     }
 
-    # As of November 2018, NSPower redirects their web traffic to HTTPS. However we've been
-    # getting SSL errors in Python, even though loading these URLs in Firefox 63 gives no errors.
-    # This isn't confidential or security-sensitive data, so ignore the SSL errors.
-    mix_url = 'https://www.nspower.ca/system_report/today/currentmix.json'
-    mix_data = requests_obj.get(mix_url, verify=False).json()
-
-    load_url = 'https://www.nspower.ca/system_report/today/currentload.json'
-    load_data = requests_obj.get(load_url, verify=False).json()
+    mix_url = 'https://www.nspower.ca/library/CurrentLoad/CurrentMix.json'
+    mix_data = requests_obj.get(mix_url).json()
+
+    load_url = 'https://www.nspower.ca/library/CurrentLoad/CurrentLoad.json'
+    load_data = requests_obj.get(load_url).json()
 
     production = []
     imports = []


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Negative Hydro values (in EIA, US) cause parser to fail
When scraping historic data, I've noticed that negative hydro values are sometimes returned by EIA. For example, for US-TN 2018-09-23 6am local time, there was hydro production reported of -144.

I am not sure if this means we should just always move negative hydro production values over to storage, or if for each BA we should decide if the hydro production should be 'storage' hydro or run-of-river. What you think? @systemcatch I guess you are closest to the EIA data.


Discussion:
Hi @robertahunt, the EIA have a useful information section about their data at https://www.eia.gov/beta/electricity/gridmonitor/about

Quoting from that page;

> Net generation

> EIA imputes for anomalous values for total net generation and net generation by energy source if the value is missing or reported as negative, zero, or at least 1.5 times greater than the maximum of past total net generation values reported by that BA with the following exceptions. Small negative net generation and zero net generation values are accepted for generation-only balancing authorities (See the Generation-only balancing authorities section above) and for City of Homestead (HST), Duke Energy Progress West (CPLW), and the Utilities Commission of New Smyrna Beach (NSB) since they may not be running their generators during all hours. Hydro net generation may also have negative values due to pumped storage, although EIA has asked balancing authorities to count pumped storage negative net generation as demand instead.

I think you're safe to assume that negative hydro values are pumped storage, given how many BA's there are trying to determine output type for each sounds like a nightmare.

Resolving patch:
diff --git a/parsers/EIA.py b/parsers/EIA.py
--- a/parsers/EIA.py
+++ b/parsers/EIA.py
@@ -93,12 +93,18 @@ def fetch_production_mix(zone_key, session=None, target_datetime=None, logger=No
         if not mix:
             continue
         for point in mix:
-            point.update({
-                'production': {type: point.pop('value')},
-                'storage': {},  # required by merge_production_outputs()
-            })
-
-            #replace small negative solar values (>-5) with 0s
+            if type == 'hydro' and point['value'] < 0:
+                point.update({
+                    'production': {},# required by merge_production_outputs()
+                    'storage': {type: point.pop('value')},
+                })
+            else:
+                point.update({
+                    'production': {type: point.pop('value')},
+                    'storage': {},  # required by merge_production_outputs()
+                })
+
+            #replace small negative values (>-5) with 0s This is necessary for solar
             point = validate(point, logger=logger, remove_negative=True)
         mixes.append(mix)
 


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Interconnectors IT-GR and IT-ME
The two interconnectors are available on the ENTSO-E transparency portal, but not on the map.

The IT-GR one, unlike the rest of the Greece data, seems to have a <2h delay, so it should be realtime enough, while the recently added IT-ME one started to return data just today [2019-12-28] (for now it sits at '0', but before it was N/A or n/e), so it may be operative soon.

The IT-ME interconnector, if added, should have a capacity of 600MW on both directions, as per https://www.terna.it/en/media/news-events/italy-montenegro-invisible-electrical-bridge


Discussion:
The Montenegro interconnector have started to report real time data today [2020-10-07], while until yesterday it seemed to have a 1-day delay.
Hi @Juma7C9 thanks for bringing this to our attention! Is this something you are comfortable trying to adjust yourself in the codebase? Would you need any pointers from us? If you cannot / don't have time, no worries! We will pick this up when we get a chance.
Cheers,
Roberta
Another potential data source: http://cges.me/en/system-live/web-scada

Resolving patch:
diff --git a/parsers/ENTSOE.py b/parsers/ENTSOE.py
--- a/parsers/ENTSOE.py
+++ b/parsers/ENTSOE.py
@@ -158,7 +158,10 @@
     'DK-DK2->SE': [ENTSOE_DOMAIN_MAPPINGS['DK-DK2'],
                    ENTSOE_DOMAIN_MAPPINGS['SE-SE4']],
     'FR-COR->IT-CNO': ['10Y1001A1001A893', ENTSOE_DOMAIN_MAPPINGS['IT-CNO']],
-    'GR->IT-SO': ['10YGR-HTSO-----Y', ENTSOE_DOMAIN_MAPPINGS['IT-BR']],
+    'GR->IT-SO': [ENTSOE_DOMAIN_MAPPINGS['GR'],
+                  ENTSOE_DOMAIN_MAPPINGS['IT-SO']],
+    'IT-CSO->ME': [ENTSOE_DOMAIN_MAPPINGS['IT'],
+                   ENTSOE_DOMAIN_MAPPINGS['ME']],
     'NO-NO3->SE': [ENTSOE_DOMAIN_MAPPINGS['NO-NO3'],
                    ENTSOE_DOMAIN_MAPPINGS['SE-SE2']],
     'NO-NO4->SE': [ENTSOE_DOMAIN_MAPPINGS['NO-NO4'],


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Exchange to CL-SING needs to be renamed to CL-SEN
![image](https://user-images.githubusercontent.com/13189214/76600946-39145900-6508-11ea-98df-b73ff0706ebc.png)

CL-SING was merged into CL-SEN, but it seems like we didn't fix that in all the exchanges json. Flagging this here so it isn't forgotten.


Discussion:

Resolving patch:
diff --git a/parsers/AR.py b/parsers/AR.py
--- a/parsers/AR.py
+++ b/parsers/AR.py
@@ -599,7 +599,7 @@
 
 cammesa_url = 'http://portalweb.cammesa.com/default.aspx'
 
-tie_mapping = {'CL-SING': "position:absolute; top:349; left:585",
+tie_mapping = {'CL-SEN': "position:absolute; top:349; left:585",
                'PY': "position:absolute; top:67; left:649",
                'UY_1': "position:absolute; top:203; left:533",
                'UY_2': "position:absolute; top:226; left:515"
@@ -966,7 +966,7 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
 
     sortedcodes = '->'.join(sorted([zone_key1, zone_key2]))
 
-    if sortedcodes == 'AR->CL-SING':
+    if sortedcodes == 'AR->CL-SEN':
         base_url = 'http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Unifilar de NOA&'
     else:
         base_url = 'http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Tensiones y Flujos de Potencia&'
@@ -981,8 +981,8 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
         flow = first_tie + second_tie
     elif sortedcodes == 'AR->PY':
         flow = tie_finder(exchange_url, 'PY', s)
-    elif sortedcodes == 'AR->CL-SING':
-        flow = tie_finder(exchange_url, 'CL-SING', s)
+    elif sortedcodes == 'AR->CL-SEN':
+        flow = tie_finder(exchange_url, 'CL-SEN', s)
     else:
         raise NotImplementedError('This exchange is not currently implemented')
 
@@ -1007,5 +1007,5 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
     print(fetch_exchange('AR', 'PY'))
     print('fetch_exchange(AR, UY) ->')
     print(fetch_exchange('AR', 'UY'))
-    print('fetch_exchange(AR, CL-SING) ->')
-    print(fetch_exchange('AR', 'CL-SING'))
+    print('fetch_exchange(AR, CL-SEN) ->')
+    print(fetch_exchange('AR', 'CL-SEN'))


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
Chile inconsistency between live and historical data acquisition
Our systems use historical data to train forecasts. When run in production, those forecasts require live data. Therefore, the two datafeeds need to be consistent.
I think the Chile parser uses two different datafeeds that aren't consistent, and thus our forecasts might be inconsistent.
If in doubt, we should only implement the real-time feed to avoid our database having both data mixed.


Discussion:
Can you post an example to let us know how inconsistent the data is? Maybe there's a bug in one of the parsers?
> Can you post an example to let us know how inconsistent the data is? Maybe there's a bug in one of the parsers?

I don't know for sure that the data is different, but from looking at the code, it uses two different URLs so potentially the data is different. I created the issue to get some certainty around this.

Resolving patch:
diff --git a/parsers/CL.py b/parsers/CL.py
--- a/parsers/CL.py
+++ b/parsers/CL.py
@@ -103,6 +103,17 @@ def production_processor_historical(raw_data):
 
     ordered_data = sorted(combined.values(), key=itemgetter("datetime"))
 
+    # For consistency with live API, hydro and geothermal must be squeezed into unknown
+    for datapoint in ordered_data:
+        if 'unknown' not in datapoint:
+            datapoint['unknown'] = 0
+        if 'hydro' in datapoint:
+            datapoint['unknown'] += datapoint['hydro']
+            del datapoint['hydro']
+        if 'geothermal' in datapoint:
+            datapoint['unknown'] += datapoint['geothermal']
+            del datapoint['geothermal']
+
     return ordered_data
 
 


Test patch:
Repository: electricitymaps/electricitymaps-contrib

Issue:
US-PJM Parser Returns Future Date After Midnight
**Steps to reproduce**

This only works shortly after midnight in the New York time zone (UTC-4) - reproduced at ~1am NY time, last value was from 10pm the day before.

* Run `fetch_production` and check the datetime of the result. It will have something like 10pm on the current day (e.g. in the future)

**Cause**

- https://github.com/tmrowco/electricitymap-contrib/blob/d7c4c2ece99a9ec5e9b67323e4bd2a40857ad38d/parsers/US_PJM.py#L90
- https://www.kite.com/python/docs/arrow.Arrow.shift 

> `Returns a new :class:`Arrow <arrow.arrow.Arrow>` object with attributes updated according to inputs`

- does **not** mutate the passed object


Discussion:

Resolving patch:
diff --git a/parsers/US_PJM.py b/parsers/US_PJM.py
--- a/parsers/US_PJM.py
+++ b/parsers/US_PJM.py
@@ -87,7 +87,7 @@ def extract_data(session=None):
     if arr_dt > future_check:
         # Generation mix lags 1-2hrs behind present.
         # This check prevents data near midnight being given the wrong date.
-        arr_dt.shift(days=-1)
+        arr_dt = arr_dt.shift(days=-1)
 
     dt = arr_dt.floor('minute').datetime
 


Test patch: