Repository: electricitymaps/electricitymaps-contrib

Issue:
Check import/export directions for Northern Ireland
We had a PR #1669 that dealt with the direction of exchanges with Northern Ireland (GB-NIR).
Today's graph for NIR looks very suspicious to me:
![image](https://user-images.githubusercontent.com/25743609/54869982-b9dec000-4da0-11e9-8fb8-f1704b2e56b7.png)
Simultaneous imports towards NIR should result in a smooth load curve or in a reaction of the production, not in a weird up and down.

Here is an overlay of the forecasted load over production:
![image](https://user-images.githubusercontent.com/25743609/54870328-3ecbd880-4da5-11e9-8fff-ac6e7e5c8f7a.png)


Another evidence for the issue is the production mix for Ireland:
![image](https://user-images.githubusercontent.com/25743609/54870335-6d49b380-4da5-11e9-9003-82066e17ed00.png)

The sum of all sources is too low to cover the load according to the data on ENTSOE:
![image](https://user-images.githubusercontent.com/25743609/54870316-2956ae80-4da5-11e9-97e7-4c0388e32838.png)

Clearly, there are a few hundred megawatts missing. I assume these should come from UK to NIR, are transferred through NIR and are passed on to IR.

Please have another look at the directions!
Kind regards, Alex


Discussion:
So I had another look at the related data and I think I found the main bug :)

1.) The data for the moyle-interconnector in fetch_exchange is problematic. 
The csv behind the data (get it at http://ws.soni.ltd.uk/DownloadCentre/aspx/MoyleTie.aspx) is either showing a negative value (=import to NIR) or 0 for Moyle, but never (!) a positive value.
This means, on the map, the power flow is only coming from GB, there are no exports from NIR to GB, which is clearly wrong.
However, the complete data for Moyle, including the positive export values, are found in the "wind_url" accessed by the parser for wind production: http://ws.soni.ltd.uk/DownloadCentre/aspx/SystemOutput.aspx

The red data from "Moyle and TieLines" shouldn't be used anymore (0 or negativ). We should stick to the Moyle data coming from the ["SystemOutput" csv](http://ws.soni.ltd.uk/DownloadCentre/aspx/SystemOutput.aspx) (covering both directions! ;) ). Please note, the direction of the values is inversed, this needs to be adressed correctly in the parser. Positive means import to NIR from GB through Moyle. Negative means export to GB from NIR.
![image](https://user-images.githubusercontent.com/25743609/56204088-e657c600-6046-11e9-9195-20bb3236b2dd.png)

2.) Actually, between NIR and IR, there are 3 interconnectors.
The main interconnector is Tandragee(NIR)-Louth(IR) on the 275kV level (2 parallel systems). 
There are 2 more 110 kV lines between the west of NIR and IR. These two interconnectors Enniskillen(NIR)-Corraclassy(IR) and Strabane(NIR)-Letterkenny(IR) "support the grid on a local level".
https://issuu.com/designtactics/docs/soni_north_south_interconnector_-_a/5
The parser is accessing the ["Moyle and TieLines"-url](http://ws.soni.ltd.uk/DownloadCentre/aspx/MoyleTie.aspx) and correctly takes the sum of all three interconnections. However, from what I have observed, that data is lacking the two 110kV-tie-lines (data for them is empty in the csv right now). 
There is another column in the "SystemOutput" csv, which is named "Tie_Lines_MW". This one sums the data for all three NIR<->IR interconnectors, and not only the 275kV line (unlike the "Moyle and Tie Lines" csv).
Here you can see that there is a small difference between the 275kV-data and the total tie-line value:
![image](https://user-images.githubusercontent.com/25743609/56205448-194f8900-604a-11e9-94ed-2cfddbaf4498.png)
So I suggest we should stick with the "Tie_Lines_MW" from SystemOutput.csv instead of what is used in fetch_exchange now.

3.) I assumed there is a wrong direction for one of the interconnectors. I haven't yet looked into the parser to find the line which may cause it, but I'll try to figure it out. I still assume it's the direction for the tie lines between NIR<->IR which is inversed.
____________________________

To get an idea of the correct direction of flows from my assumptions:

![image](https://user-images.githubusercontent.com/25743609/56205562-73504e80-604a-11e9-80a7-68ff0d47777e.png)
- "Total_NI_Generated_MW" = Load, green surface.
- "Tie_Lines_MW" = exchange NIR<->IR, with negative = "export from NIR to IR" and positive "import to NIR from IR"
- "Total_Moyle_Load_MW" = exchange NIR<->GB, with negative = "export from NIR to GB" and positive "import to NIR from GB" (I cross-checked with ENTSOE data)
-  surplusses (everything above green surface) are exported to IR (dark red in negative area) or GB (yellow in negative area)

I'll try to implement the suggested improvements ;)

Resolving patch:
diff --git a/parsers/GB_NIR.py b/parsers/GB_NIR.py
--- a/parsers/GB_NIR.py
+++ b/parsers/GB_NIR.py
@@ -13,11 +13,16 @@
 
 from .lib.validation import validate
 
-thermal_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/FuelMix.aspx'
-wind_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/SystemOutput.aspx'
-exchange_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/MoyleTie.aspx'
-# Positive values represent exports to Northern Ireland.
-# Negative value represent imports from Northern Ireland.
+production_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/FuelMix.aspx'
+exchange_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/SystemOutput.aspx'
+# Positive values in the .csv represent imports to Northern Ireland from GB / IR.
+# Negative values in the .csv represent exports from Northern Ireland to GB / IR.
+
+## exchange_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/MoyleTie.aspx' ## see comment below
+## Old exchange_url was used for exchanges, but it provided incomplete data.
+## "Total_Moyle_Load_MW" was showing "0" when exporting to GB. Tie-line data was missing for the two 110kV Lines "Enniskillen(NIR)-Corraclassy(IR)" and "Strabane(NIR)-Letterkenny(IR)"
+## Exchanges are now based on data with better quality provided in "SystemOutput.csv" under the new exchange_url = 'http://ws.soni.ltd.uk/DownloadCentre/aspx/SystemOutput.aspx'
+
 
 
 def get_data(url, target_datetime, session=None):
@@ -90,30 +95,17 @@ def add_default_tz(timestamp):
     return modified_timestamp
 
 
-def create_thermal_df(text_data):
+def create_production_df(text_data):
     """
     Turns thermal csv data into a usable dataframe.
     """
 
     cols_to_use = [0, 1, 2, 3, 4, 5]
-    df_thermal = pd.read_csv(StringIO(text_data),
+    df_production = pd.read_csv(StringIO(text_data),
                              usecols=cols_to_use)
-    df_thermal.fillna(0.0, inplace=True)
-
-    return df_thermal
+    df_production.fillna(0.0, inplace=True)
 
-
-def create_wind_df(text_data):
-    """
-    Turns wind csv data into a usable dataframe.
-    """
-
-    cols_to_use = [0, 1]
-    df_wind = pd.read_csv(StringIO(text_data),
-                          usecols=cols_to_use)
-    df_wind.fillna(0.0, inplace=True)
-
-    return df_wind
+    return df_production
 
 
 def create_exchange_df(text_data):
@@ -127,7 +119,7 @@ def create_exchange_df(text_data):
     return df_exchange
 
 
-def thermal_processor(df):
+def production_processor(df):
     """
     Creates quarter hour datapoints for thermal production.
     Returns a list.
@@ -136,26 +128,12 @@ def thermal_processor(df):
     datapoints = []
     for index, row in df.iterrows():
         snapshot = {}
-        snapshot['datetime'] = row['TimeStamp']
+        snapshot['datetime'] = add_default_tz(parser.parse(row['TimeStamp'],
+                                                           dayfirst=True))
         snapshot['gas'] = row['Gas_MW']
         snapshot['coal'] = row['Coal_MW']
         snapshot['oil'] = row['Distillate_MW'] + row['Diesel_MW']
-        datapoints.append(snapshot)
-
-    return datapoints
-
-
-def wind_processor(df):
-    """
-    Creates quarter hour datapoints for wind production.
-    Returns a list.
-    """
-
-    datapoints = []
-    for index, row in df.iterrows():
-        snapshot = {}
-        snapshot['datetime'] = row['TimeStamp']
-        snapshot['wind'] = row['Total_Wind_Generated_MW']
+        snapshot['wind'] = row['Wind_MW']
         if snapshot['wind'] > -20:
             snapshot['wind'] = max(snapshot['wind'], 0)
         datapoints.append(snapshot)
@@ -193,9 +171,7 @@ def IE_processor(df):
         snapshot = {}
         snapshot['datetime'] = add_default_tz(parser.parse(row['TimeStamp'],
                                                            dayfirst=True))
-        netFlow = (row['Total_Str_Let_Load_MW'] +
-                   row['Total_Enn_Cor_Load_MW'] +
-                   row['Total_Tan_Lou_Load_MW'])
+        netFlow = -1* row['Tie_Lines_MW']
         snapshot['netFlow'] = netFlow
         snapshot['source'] = 'soni.ltd.uk'
         snapshot['sortedZoneKeys'] = 'GB-NIR->IE'
@@ -204,27 +180,6 @@ def IE_processor(df):
     return datapoints
 
 
-def merge_production(thermal_data, wind_data):
-    """
-    Joins thermal and wind production data on shared datetime key.
-    Returns a list.
-    """
-
-    total_production = thermal_data + wind_data
-
-    # Join thermal and wind dicts on 'datetime' key.
-    d = defaultdict(dict)
-    for elem in total_production:
-        d[elem['datetime']].update(elem)
-
-    joined_data = sorted(d.values(), key=itemgetter("datetime"))
-
-    for datapoint in joined_data:
-        datapoint['datetime'] = add_default_tz(parser.parse(datapoint['datetime'], dayfirst=True))
-
-    return joined_data
-
-
 def fetch_production(zone_key='GB-NIR', session=None, target_datetime=None,
                      logger=logging.getLogger(__name__)):
     """
@@ -256,17 +211,13 @@ def fetch_production(zone_key='GB-NIR', session=None, target_datetime=None,
         }
     """
 
-    thermal_data = get_data(thermal_url, target_datetime)
-    wind_data = get_data(wind_url, target_datetime)
-    thermal_df = create_thermal_df(thermal_data)
-    wind_df = create_wind_df(wind_data)
-    thermal = thermal_processor(thermal_df)
-    wind = wind_processor(wind_df)
-    merge = merge_production(thermal, wind)
+    production_data = get_data(production_url, target_datetime)
+    production_df = create_production_df(production_data)
+    production = production_processor(production_df)
 
     production_mix_by_quarter_hour = []
 
-    for datapoint in merge:
+    for datapoint in production:
         production_mix = {
             'zoneKey': zone_key,
             'datetime': datapoint.get('datetime', 0.0),
@@ -314,7 +265,6 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
 
 if __name__ == '__main__':
     """Main method, never used by the Electricity Map backend, but handy for testing."""
-
     print('fetch_production() ->')
     print(fetch_production())
     print('fetch_exchange(GB-NIR, GB) ->')


Test patch: