Repository: electricitymaps/electricitymaps-contrib

Issue:
Change data source for UK-GB (5-min data)
I've seen this happen frequently in the last weeks: the wind energy production on ENTSO-E for the UK would drop and leave a very spiky and obviously wrong graph for the UK-GB zone.

![image](https://user-images.githubusercontent.com/25743609/39520683-e3e0788e-4e0b-11e8-8a36-370ec84be0a7.png)

This issue got much worse in the last three days. Here is what it looked like for 1st of May [on ENTSO-E](https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show?name=&defaultValue=false&viewType=GRAPH&areaType=BZN&atch=false&datepicker-day-offset-select-dv-date-from_input=D&dateTime.dateTime=01.05.2018+00:00|CET|DAYTIMERANGE&dateTime.endDateTime=01.05.2018+00:00|CET|DAYTIMERANGE&area.values=CTY|GB!BZN|10YGB----------A&productionType.values=B17&productionType.values=B18&productionType.values=B19&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2)):

![image](https://user-images.githubusercontent.com/25743609/39520636-bbc15ea4-4e0b-11e8-8670-fbe0f6450a43.png)

Offshore generation would almost completely disappear and onshore wind would drop by 30-40%.

Other sources, like these dashboards don't seem to be affected by that data quality issue.: 
http://electricinsights.co.uk/#/dashboard?&_k=dxtgvl 
http://gridwatch.co.uk/
http://grid.iamkate.com/

So maybe it's time to switch to data from ELEXON, which have a super short reporting interval of 5 minutes for both generation and all exchanges. Seems like there is a delay of 1 hour:

https://www.bmreports.com/bmrs/?q=generation/fueltype

![image](https://user-images.githubusercontent.com/25743609/39522025-3fa9305c-4e11-11e8-994b-e66d7465e0fe.png)

Check out the XML and CSV links on the site that'll lead you to the data.
Historical data available, too!


Discussion:
ELEXON is great! I have dealt with this source in my day job. Let me know where I can help out.
Yeah, true, the closer to the source is always the best...! and GB is soon going to able to be broken down by DSO's regions in National Grid / Elexon systems apparently. Is there also a 1h delay is their API?
We should collect this source in addition to entsoe, there may be some interesting differences. The xml looks like a better choice as the csv data doesn't have column headers?
CSV will be easier to parse though, so I'd suggest going the CSV way.
I'm going to code this up this evening, if nobody else has started.
I far prefer parsing CSVs. The only thing to note is you are relying on the fuel column order outlined in the documentation being and staying correct as well as applying small edits when new fuels are added. Not sure if this will happen again but I recall Biomass was split out of one of the other fuel types around November '17.
@lorrieq we could check the number of columns in the csv and log a warning if it's changed from the current number.
Hey all, I'm almost ready to submit an initial PR for this but wanted to check a few things.

1) To confirm, the datetimes in the parsers output should be in UTC right?

2) The datetime for each observation. Should this be the published time minus 5 minutes? In the example, the first observation for SP 28 which is 12:30GMT has a published time of 12:35GMT. My interpretation is the parser give this observation a datetime of 12:30GMT, the observation with 12:40GMT publish time should have a datetime of 12:35GMT and so on.

3) The exchanges. I don't think there is any way to tell the netFlow, there is no reference to the direction of flow, just a total MW value which probably is an aggregation of import/export?

4) How are we dealing with multiple parsers per zone? I don't see any examples of this to date. Maybe allow the parsers dictionary values map to a list (as shown in image) rather than single string at the moment? I presume this would need some change to the backend code.
<img width="508" alt="screen shot 2018-05-06 at 15 02 15" src="https://user-images.githubusercontent.com/6525086/39674074-7fffd57c-513e-11e8-997e-6fe7d73c159c.png">

1. datetime object with a timezone is fine.

2. How exactly is the settlement period used?

3. The exchanges on the generation by fuel page seem to ignore negative values, but on https://www.bmreports.com/bmrs/?q=generation/avghalfhourIC they don't.

4. I think that's the way to go but @corradio is the one who will make the changes to the backend.

Resolving patch:
diff --git a/parsers/ELEXON.py b/parsers/ELEXON.py
new file mode 100644
--- /dev/null
+++ b/parsers/ELEXON.py
@@ -0,0 +1,249 @@
+#!/usr/bin/env python3
+# coding=utf-8
+
+"""
+Parser that uses the ELEXON API to return the following data types.
+
+Production
+Exchanges
+
+Documentation:
+https://www.elexon.co.uk/wp-content/uploads/2017/06/
+bmrs_api_data_push_user_guide_v1.1.pdf
+"""
+
+import os
+import arrow
+import logging
+import requests
+import datetime as dt
+import pandas as pd
+from io import StringIO
+
+ELEXON_ENDPOINT = 'https://api.bmreports.com/BMRS/{}/v1'
+
+REPORT_META = {
+    'B1620': {
+        'expected_fields': 13,
+        'skiprows': 5
+    },
+    'INTERFUELHH': {
+        'expected_fields': 7,
+        'skiprows': 0
+    }
+}
+
+# 'hydro' key is for hydro production
+# 'hydro storage' key is for hydro storage
+RESOURCE_TYPE_TO_FUEL = {
+    'Biomass': 'biomass',
+    'Fossil Gas': 'gas',
+    'Fossil Hard coal': 'coal',
+    'Fossil Oil': 'oil',
+    'Hydro Pumped Storage': 'hydro storage',
+    'Hydro Run-of-river and poundage': 'hydro',
+    'Nuclear': 'nuclear',
+    'Solar': 'solar',
+    'Wind Onshore': 'wind',
+    'Wind Offshore': 'wind',
+    'Other': 'unknown'
+}
+
+EXCHANGES = {
+    'FR->GB': 3,
+    'GB-NIR->IE': 4,
+    'GB->NL': 5,
+    'GB->IE': 6
+}
+
+
+def query_ELEXON(report, session, params):
+    if 'ELEXON_TOKEN' not in os.environ:
+        raise Exception('No ELEXON_TOKEN found! Please add it to secrets.env!')
+    params['APIKey'] = os.environ['ELEXON_TOKEN']
+    return session.get(ELEXON_ENDPOINT.format(report), params=params)
+
+
+def query_exchange(session, target_datetime=None):
+    if target_datetime is None:
+        target_datetime = dt.date.today()
+
+    from_date = (target_datetime - dt.timedelta(days=1)).strftime('%Y-%m-%d')
+    to_date = target_datetime.strftime('%Y-%m-%d')
+
+    params = {
+        'FromDate': from_date,
+        'ToDate': to_date,
+        'ServiceType': 'csv'
+    }
+    response = query_ELEXON('INTERFUELHH', session, params)
+    return response.text
+
+
+def query_production(session, target_datetime=None):
+    if target_datetime is None:
+        target_datetime = dt.datetime.now()
+
+    # we can only fetch one date at a time.
+    # if target_datetime is first 30 minutes of the day fetch the day before.
+    # otherwise fetch the day of target_datetime.
+    if target_datetime.time() <= dt.time(0, 30):
+        settlement_date = target_datetime.date() - dt.timedelta(1)
+    else:
+        settlement_date = target_datetime.date()
+
+    params = {
+        'SettlementDate': settlement_date.strftime('%Y-%m-%d'),
+        'Period': '*',
+        'ServiceType': 'csv'
+    }
+    response = query_ELEXON('B1620', session, params)
+    return response.text
+
+
+def parse_exchange(zone_key1, zone_key2, csv_text, target_datetime=None,
+                   logger=logging.getLogger(__name__)):
+    if not csv_text:
+        return None
+
+    report = REPORT_META['INTERFUELHH']
+
+    sorted_zone_keys = sorted([zone_key1, zone_key2])
+    exchange = '->'.join(sorted_zone_keys)
+    data_points = list()
+    lines = csv_text.split('\n')
+
+    # check field count in report is as expected
+    field_count = len(lines[1].split(','))
+    if field_count != report['expected_fields']:
+        raise ValueError(
+            'Expected {} fields in INTERFUELHH report, got {}'.format(
+                report['expected_fields'], field_count))
+
+    for line in lines[1:-1]:
+        fields = line.split(',')
+
+        # settlement date / period combinations are always local time
+        date = dt.datetime.strptime(fields[1], '%Y%m%d').date()
+        settlement_period = int(fields[2])
+        datetime = datetime_from_date_sp(date, settlement_period)
+
+        data = {
+            'sortedZoneKeys': exchange,
+            'datetime': datetime,
+            'source': 'bmreports.com'
+        }
+
+        # positive value implies import to GB
+        multiplier = -1 if 'GB' in sorted_zone_keys[0] else 1
+        data['netFlow'] = float(fields[EXCHANGES[exchange]]) * multiplier
+        data_points.append(data)
+
+    return data_points
+
+
+def parse_production(csv_text, target_datetime=None,
+                     logger=logging.getLogger(__name__)):
+    if not csv_text:
+        return None
+
+    report = REPORT_META['B1620']
+
+    # create DataFrame from slice of CSV rows
+    df = pd.read_csv(StringIO(csv_text), skiprows=report['skiprows'] - 1)
+
+    # check field count in report is as expected
+    field_count = len(df.columns)
+    if field_count != report['expected_fields']:
+        raise ValueError(
+            'Expected {} fields in B1620 report, got {}'.format(
+                report['expected_fields'], len(df.columns)))
+
+    # filter out undesired columns
+    df = df.iloc[:-1, [7, 8, 9, 4]]
+
+    df['Settlement Date'] = df['Settlement Date'].apply(
+        lambda x: dt.datetime.strptime(x, '%Y-%m-%d'))
+    df['Settlement Period'] = df['Settlement Period'].astype(int)
+    df['datetime'] = df.apply(lambda x: datetime_from_date_sp(
+        x['Settlement Date'], x['Settlement Period']), axis=1)
+
+    # map from report fuel names to electricitymap fuel names
+    fuel_column = 'Power System Resource  Type'
+    df[fuel_column] = df[fuel_column].apply(lambda x: RESOURCE_TYPE_TO_FUEL[x])
+
+    # loop through unique datetimes and create each data point
+    data_points = list()
+    for time in pd.unique(df['datetime']):
+        time_df = df[df['datetime'] == time]
+
+        data_point = {
+            'zoneKey': 'GB',
+            'datetime': time.to_pydatetime(),
+            'source': 'bmreports.com',
+            'production': dict(),
+            'storage': dict()
+        }
+
+        for row in time_df.iterrows():
+            fields = row[1].to_dict()
+            fuel = fields[fuel_column]
+            quantity = fields['Quantity']
+
+            # check if storage value and if so correct key
+            if 'storage' in fuel:
+                fuel_key = fuel.replace('storage', '').strip()
+                # ELEXON storage is negative when storing and positive when
+                # discharging (the opposite to electricitymap)
+                data_point['storage'][fuel_key] = quantity * -1
+            else:
+                # if/else structure allows summation of multiple quantities
+                # e.g. 'Wind Onshore' and 'Wind Offshore' both have the
+                # key 'wind' here.
+                if fuel in data_point['production'].keys():
+                    data_point['production'][fuel] += quantity
+                else:
+                    data_point['production'][fuel] = quantity
+
+        data_points.append(data_point)
+
+    return data_points
+
+
+def datetime_from_date_sp(date, sp):
+    datetime = arrow.get(date).shift(minutes=30 * (sp - 1))
+    return datetime.replace(tzinfo='Europe/London').datetime
+
+
+def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None,
+                   logger=logging.getLogger(__name__)):
+    session = session or requests.session()
+    response = query_exchange(session, target_datetime)
+    data = parse_exchange(zone_key1, zone_key2, response, target_datetime,
+                          logger)
+    return data
+
+
+def fetch_production(zone_key='GB', session=None, target_datetime=None,
+                     logger=logging.getLogger(__name__)):
+    session = session or requests.session()
+    response = query_production(session, target_datetime)
+    data = parse_production(response, target_datetime, logger)
+    return data
+
+
+if __name__ == '__main__':
+    """Main method, never used by the Electricity Map backend, but handy 
+    for testing."""
+
+    print('fetch_production() ->')
+    print(fetch_production())
+
+    print('fetch_exchange(FR, GB) ->')
+    print(fetch_exchange('FR', 'GB'))
+
+    print('fetch_exchange(GB, IE) ->')
+    print(fetch_exchange('GB', 'IE'))
+
+    print('fetch_exchange(GB, NL) ->')
+    print(fetch_exchange('GB', 'NL'))


Test patch: