Repository: electricitymaps/electricitymaps-contrib

Issue:
Add static carbon intensity for NL exports



Discussion:
Talked to Dr. Martien Visser from Entrance (cc @alixunderplatz)
> To make it easy: the number for the last 12 months (April 2017 - March 2018) is 437 gr/kWh.

I think that's a number we can use!
@corradio I still have some concerns about spoiling the export intensity. NL is kind of different from the other areas where a hardcoded intensity is used. These zones had a more "predictable", "not-so-diversified" or "fluctuating-RES" generation portfolio.

Opposed to this, The Netherlands
* have a considerable amount of fluctuating solar and especially wind
* recieve & transport a large amount of power. In 2017:
  * the physical imports from DE were 14.2 TWh, while exports to DE only were 0.42 TWh
  * the physical imports from NO were 5 TWH, while exports to NO only were 0.11 TWh 
    * DE and NO are sources for the NL mix. NO's intensity is almost static, DE is fluctuating heavily.
  * the physical imports from BE only were 1.6 TWh, while exports to BE were 9.6 TWh
  * the physical imports from GB only were 0.19 TWh, while exports to GB were 7.2 TWh
    * BE and GB are recievers of power from the NL mix

Yearly numbers for physical power flows are from [energy-charts](https://energy-charts.de/exchange_de.htm), who take them from ENTSO-E.

![image](https://user-images.githubusercontent.com/25743609/38298812-bd562052-37f8-11e8-87e6-53ab745a0737.png)

Does the static value of 437 g/kWh include imports from NO and DE, or is it for the domestic generation only? Right now, about 4 GW come in from DE, which makes up 20-25% of the electricity mix of NL. Even if this is just a temporary solution, I am wondering how strong the carbon intensity will deviate from the static figure. It will have a limited influence on GB's intensity, because transmission capacity is limited to 1 GW. However, for for BE, where one could also observe exports from NL of around 3 GW over longer periods, it will have some significant influence on the mix and carbon intensity.
It's domestic generation only, so indeed it is a very crude approximation. You are totally right.
How do you suggest we proceed? Correct it with the average flows and the intensities of neighbours?
That's really a tough question.
The only thing that seems possible at the moment is combining total generation [from TenneT](https://www.tennet.eu/nl/elektriciteitsmarkt/data-dashboard/productie/) (even though it is not always reliably available) with the static intensity and putting it to the "unknown" category. This way, at least the imports would have their influence on the average domestic carbon intensity. If it's possible, I suggest leaving NL in grey and only consider this "synthetic mix" for the export intensity.

![image](https://user-images.githubusercontent.com/25743609/38304144-aa34746e-3808-11e8-9759-0d95fbae9fff.png)

Unfortunately, the recent generation is displayed as "-" at the moment. Maybe that's a temporary issue.

![image](https://user-images.githubusercontent.com/25743609/38305122-fbd130c0-380b-11e8-9a36-17c047aa450a.png)

Actually you don't have to put it all under "unknown" because renewables production is available from:
http://energieopwek.nl

Separated in biomass, wind and solar.
@PaulCornelissen yeah, that site had also been mentioned in the main issue #78.

As far as I can recall, these figures are estimates and would "spoil" the mix with the static carbon intensity in return, because they are already included in that static value that is mentioned above.
In addition, in the biogas category, they only seem to consider generation from biogas and not from other biomass like wood chips (I think there were projects to use wood chips as fuel in coal power plants, similar to Denmark?!) or waste.
A more detailed answer from Dr. Visser on the (April 2017 - March 2018) annual NL average production (MWh) is: 
> 30% coal, 20% cogeneration (gas) and 28% gas-fired, renewables (14%), coke-oven gas (5%), nuclear (4%). 

We shall be able to compute our own carbon-intensity for an "unknown" category which would include coal, cogen, gas, coke and nuclear. And use biomass, wind and solar from Tennet. Let's try!

Resolving patch:
diff --git a/parsers/ENTSOE.py b/parsers/ENTSOE.py
--- a/parsers/ENTSOE.py
+++ b/parsers/ENTSOE.py
@@ -431,23 +431,25 @@ def datetime_from_position(start, position, resolution):
     raise NotImplementedError('Could not recognise resolution %s' % resolution)
 
 
-def parse_consumption(xml_text):
+def parse_scalar(xml_text):
     """Returns a tuple containing two lists."""
 
     if not xml_text:
         return None
     soup = BeautifulSoup(xml_text, 'html.parser')
     # Get all points
-    quantities = []
+    values = []
     datetimes = []
     for timeseries in soup.find_all('timeseries'):
         resolution = timeseries.find_all('resolution')[0].contents[0]
         datetime_start = arrow.get(timeseries.find_all('start')[0].contents[0])
         for entry in timeseries.find_all('point'):
-            quantities.append(float(entry.find_all('quantity')[0].contents[0]))
             position = int(entry.find_all('position')[0].contents[0])
-            datetimes.append(datetime_from_position(datetime_start, position, resolution))
-    return quantities, datetimes
+            value = float(entry.find_all('quantity')[0].contents[0])
+            datetime = datetime_from_position(datetime_start, position, resolution)
+            values.append(value)
+            datetimes.append(datetime)
+    return values, datetimes
 
 
 def parse_production(xml_text):
@@ -569,48 +571,6 @@ def parse_price(xml_text):
     return prices, currencies, datetimes
 
 
-def parse_generation_forecast(xml_text):
-    """Returns a tuple containing two lists."""
-
-    if not xml_text:
-        return None
-    soup = BeautifulSoup(xml_text, 'html.parser')
-    # Get all points
-    values = []
-    datetimes = []
-    for timeseries in soup.find_all('timeseries'):
-        resolution = timeseries.find_all('resolution')[0].contents[0]
-        datetime_start = arrow.get(timeseries.find_all('start')[0].contents[0])
-        for entry in timeseries.find_all('point'):
-            position = int(entry.find_all('position')[0].contents[0])
-            value = float(entry.find_all('quantity')[0].contents[0])
-            datetime = datetime_from_position(datetime_start, position, resolution)
-            values.append(value)
-            datetimes.append(datetime)
-    return values, datetimes
-
-
-def parse_consumption_forecast(xml_text):
-    """Returns a tuple containing two lists."""
-
-    if not xml_text:
-        return None
-    soup = BeautifulSoup(xml_text, 'html.parser')
-    # Get all points
-    values = []
-    datetimes = []
-    for timeseries in soup.find_all('timeseries'):
-        resolution = timeseries.find_all('resolution')[0].contents[0]
-        datetime_start = arrow.get(timeseries.find_all('start')[0].contents[0])
-        for entry in timeseries.find_all('point'):
-            position = int(entry.find_all('position')[0].contents[0])
-            value = float(entry.find_all('quantity')[0].contents[0])
-            datetime = datetime_from_position(datetime_start, position, resolution)
-            values.append(value)
-            datetimes.append(datetime)
-    return values, datetimes
-
-
 def validate_production(datapoint, logger):
     """
     Production data can sometimes be available but clearly wrong.
@@ -722,7 +682,8 @@ def fetch_consumption(zone_key, session=None, target_datetime=None,
         session = requests.session()
     domain = ENTSOE_DOMAIN_MAPPINGS[zone_key]
     # Grab consumption
-    parsed = parse_consumption(query_consumption(domain, session, target_datetime=target_datetime))
+    parsed = parse_scalar(
+        query_consumption(domain, session, target_datetime=target_datetime))
     if parsed:
         quantities, datetimes = parsed
 
@@ -762,8 +723,9 @@ def fetch_production(zone_key, session=None, target_datetime=None,
     production_hashmap = defaultdict(lambda: {})
     # Grab production
     for k in ENTSOE_PARAMETER_DESC.keys():
-        parsed = parse_production(query_production(k, domain, session,
-                                                   target_datetime=target_datetime))
+        parsed = parse_production(
+            query_production(k, domain, session,
+                             target_datetime=target_datetime))
         if parsed:
             productions, datetimes = parsed
             for i in range(len(datetimes)):
@@ -946,7 +908,8 @@ def fetch_exchange_forecast(zone_key1, zone_key2, session=None, target_datetime=
     return data
 
 
-def fetch_price(zone_key, session=None, target_datetime=None, logger=logging.getLogger(__name__)):
+def fetch_price(zone_key, session=None, target_datetime=None,
+                logger=logging.getLogger(__name__)):
     """
     Gets day-ahead price for specified zone.
     Returns a list of dictionaries.
@@ -985,7 +948,7 @@ def fetch_generation_forecast(zone_key, session=None, target_datetime=None,
         session = requests.session()
     domain = ENTSOE_DOMAIN_MAPPINGS[zone_key]
     # Grab consumption
-    parsed = parse_generation_forecast(query_generation_forecast(
+    parsed = parse_scalar(query_generation_forecast(
         domain, session, target_datetime=target_datetime))
     if parsed:
         data = []
@@ -1011,7 +974,7 @@ def fetch_consumption_forecast(zone_key, session=None, target_datetime=None,
         session = requests.session()
     domain = ENTSOE_DOMAIN_MAPPINGS[zone_key]
     # Grab consumption
-    parsed = parse_consumption_forecast(query_consumption_forecast(
+    parsed = parse_scalar(query_consumption_forecast(
         domain, session, target_datetime=target_datetime))
     if parsed:
         data = []
diff --git a/parsers/NL.py b/parsers/NL.py
new file mode 100644
--- /dev/null
+++ b/parsers/NL.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+
+import arrow
+from . import ENTSOE
+import logging
+import pandas as pd
+import requests
+
+
+def fetch_production(zone_key='NL', session=None, target_datetime=None,
+                     logger=logging.getLogger(__name__)):
+    if target_datetime:
+        now = arrow.get(target_datetime, 'Europe/Paris')
+    else:
+        now = arrow.now(tz='Europe/Paris')
+
+    r = session or requests.session()
+
+    consumptions = ENTSOE.fetch_consumption(zone_key=zone_key,
+                                            session=r,
+                                            target_datetime=now,
+                                            logger=logger)
+    for c in consumptions:
+        del c['source']
+    df_consumptions = pd.DataFrame.from_dict(consumptions).set_index('datetime')
+
+    # NL has exchanges with BE, DE, NO, GB
+    exchanges = []
+    for exchange_key in ['BE', 'DE', 'NO', 'GB']:
+        zone_1, zone_2 = sorted([exchange_key, zone_key])
+        exchange = ENTSOE.fetch_exchange(zone_key1=zone_1,
+                                         zone_key2=zone_2,
+                                         session=r,
+                                         target_datetime=now,
+                                         logger=logger)
+        for e in exchange:
+            e['NL_import'] = e['netFlow'] if zone_2 == 'NL' else -1 * e['netFlow']
+            del e['source']
+        exchanges.extend(exchange or [])
+    df_exchanges = pd.DataFrame.from_dict(exchanges).set_index('datetime')
+    # Sum all exchanges to NL imports
+    df_exchanges = df_exchanges.groupby('datetime').sum()
+
+    # Fill missing values by propagating the value forward
+    df_consumptions_with_exchanges = df_consumptions.join(df_exchanges).fillna(
+        method='ffill', limit=3)  # Limit to 3 x 15min
+
+    # Load = Generation + netImports
+    # => Generation = Load - netImports
+    df_total_generations = (df_consumptions_with_exchanges['consumption'] -
+        df_consumptions_with_exchanges['NL_import'])
+
+    # Fetch all production
+    productions = ENTSOE.fetch_production(zone_key=zone_key, session=r,
+                                          target_datetime=now, logger=logger)
+
+    # Flatten production dictionaries (we ignore storage)
+    for p in productions:
+        # We here assume 0 storage
+        p['production']['coal'] = None
+        p['production']['gas'] = None
+        p['production']['nuclear'] = None
+        p['production']['biomass'] = None
+
+        p['production']['unknown'] = 0
+        Z = sum([x or 0 for x in p['production'].values()])
+        p['production']['unknown'] = df_total_generations[p['datetime']] - Z
+
+    return productions
+
+
+if __name__ == '__main__':
+    print(fetch_production())


Test patch: