Repository: electricitymaps/electricitymaps-contrib

Issue:
Rewrite ENTE.py to parse all Central American exchanges
The way the recent [ENTE.py parser](https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/ENTE.py) calculates the exchanges is really complicated and not complete for all exchanges due to mathematical uncertainties.
From what I observed, there is an easier way to do this, taking the same values from the .csv plus an extra little detail! 😃

Here is the map:
http://www.enteoperador.org/newsite/flash/SER.html
![image](https://user-images.githubusercontent.com/25743609/50530807-8b36ad80-0b01-11e9-8f66-0b9122c973c2.png)

Here is the according exchange data from the .csv:
http://www.enteoperador.org/newsite/flash/data.csv

MX<->GU | MXGUN | GU<->ES | GUESN | ES<->HO | ESHON | HO<->NI | HONIN | NI<->CR | NICRN | CR<->PA | CRPAN
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
51,8 | 0 | 207,8 | 1 | 31,3 | 1 | 31 | 1 | 33,5 | 1 | 6,4 | 1

There always is a "logical value" column right behind the exchange value indicating either "1" or "0". This will flip the arrow in the network image, so it indicates the correct direction of flow.
This could help to make the parser shorter and to implement the currently missing exchange flows! :)

______________

Here is the proof for the exchange MX<->GU and CR<->PA, which changed directions:

![image](https://user-images.githubusercontent.com/25743609/50531023-4eb88100-0b04-11e9-9aae-df085772b5bd.png)

.csv data (new):

flow | MX<->GU | MXGUN | GU<->ES | GUESN | ES<->HO | ESHON | HO<->NI | HONIN | NI<->CR | NICRN | CR<->PA | CRPAN
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
old | 51,8 | 0 | 207,8 | 1 | 31,3 | 1 | 31 | 1 | 33,5 | 1 | 6,4 | 1
new | 13,1 | 1 | 242,5 | 1 | 3,6 | 1 | 11,1 | 2 | 6,9 | 1 | 1,5 | 0

Mx<->GU is now 1 and CR<->PA is 0 and the arrow direction has changed.
_____________

Note there is a "bug" 🐛 for the exchange HO<->NI, where the direction is indicated as "2" and the arrow disappears on that second map. I assume it is an issue on the data provider side and "1" means "0" and "2" means "1". 
I think it is not problematic to skip this exchange for the beginning, since we dont have any data for Honduras (besides total load and generation, if you'd like to implement this for now! ;) ).
______________

Based on the first image, all indicated flow directions are "1". Just MX<-GU has 0 there.
Not sure which is the correct arrow direction for HO<->NI (maybe it's flipped).

There probably needs to be something like this in the code to obtain the directions which we can use to determine the direction of all flows:

```
def flowdirection(df):
    """
    Gets values for each flow direction.
    Returns a dictionary.
    """

    directions = {'GT->MX': df.iloc[0]['MXGUN'],
                        'GT->SV': df.iloc[0]['GUESN'],
                        'GT->HN': df.iloc[0]['GUHON'],
                        'HN->SV': df.iloc[0]['ESHON'],
                        'HN->NI': df.iloc[0]['HONIN'],
                        'CR->NI': df.iloc[0]['NICRN'],
                        'CR->PA': df.iloc[0]['CRPAN']}

    return directions
```
__________

Any thoughts on the new assumption for the flow direction? Was the "direction indicator" in the .csv discussed in the past and have I missed it?

@systemcatch Do you want to rewrite the parser?


Discussion:
Hey @alixunderplatz, looks like you've solved the mystery of how the ENTE map gets flow arrow directions! I think when I looked at this ages ago the logical values weren't changing which led me to discount them, but clearly they are determining direction.

I can rewrite the parser easily based on this information.
@systemcatch that would be great! :)
Yeah I was lucky enough to observe these numbers changing at the right time ;)

One more info on the Honduras<->Nicaragua interconnection for you:
I saw 'HONIN' have the direction values:
- 1 with arrow pointing towards HN
- 2 without any arrow
and today I just saw
- 0 (arrow pointing towards NI)

![image](https://user-images.githubusercontent.com/25743609/50540950-f03ee180-0b9b-11e9-8de2-8d42c941e5db.png)


Resolving patch:
diff --git a/parsers/ENTE.py b/parsers/ENTE.py
--- a/parsers/ENTE.py
+++ b/parsers/ENTE.py
@@ -6,180 +6,50 @@
 import arrow
 import pandas as pd
 
-url = 'http://www.enteoperador.org/newsite/flash/data.csv'
+# map for reference
+# http://www.enteoperador.org/newsite/flash/SER.html
+DATA_URL = 'http://www.enteoperador.org/newsite/flash/data.csv'
 
-
-def read_data():
-    """
-    Reads csv data from the url.
-    Returns a pandas dataframe.
-    """
-
-    df = pd.read_csv(url, index_col=False)
-
-    return df
+CSV_MAPPING = {'GT->MX': ('MXGU', 'MXGUN'),
+               'GT->SV': ('GUES', 'GUESN'),
+               'GT->HN': ('GUHO', 'GUHON'),
+               'HN->SV': ('ESHO', 'ESHON'),
+               # 'HN->NI': ('HONI', 'HONIN'), NOTE bug in direction
+               'CR->NI': ('NICR', 'NICRN'),
+               'CR->PA': ('CRPA', 'CRPAN')}
 
 
-def connections(df):
+def extract_exchange(df, exchange):
     """
-    Gets values for each interconnection.
-    Returns a dictionary.
+    Extracts flow value and direction for a given exchange.
+    Returns a float.
     """
+    search_values = CSV_MAPPING[exchange]
+    interconnection = df.iloc[0][search_values[0]], df.iloc[0][search_values[1]]
+    direction = direction_finder(exchange, interconnection[1])
 
-    interconnections = {'GT->MX': df.iloc[0]['MXGU'],
-                        'GT->SV': df.iloc[0]['GUES'],
-                        'GT->HN': df.iloc[0]['GUHO'],
-                        'HN->SV': df.iloc[0]['ESHO'],
-                        'HN->NI': df.iloc[0]['HONI'],
-                        'CR->NI': df.iloc[0]['NICR'],
-                        'CR->PA': df.iloc[0]['CRPA']}
+    flow = interconnection[0]*direction
 
-    return interconnections
+    return flow
 
 
-def net(df):
+def direction_finder(exchange, logic):
+    """Direction of electricity flow on ENTE map is shown by arrows.
+    These arrows are controlled by booleans in the returned data.
+    Due to differences in how ENTE and EM order exchanges (e.g. ESHO vs HN->SV)
+    sometimes multiplying by -1 is needed.
+    Returns an integer.
     """
-    Gets net production values for each country.  Uses system totals for Mexico.
-    Returns a dictionary.
-    """
-
-    net_production = {}
-    net_production['GT'] = df.iloc[0]['GENGUA'] - df.iloc[0]['DEMGUA']
-    net_production['SV'] = df.iloc[0]['GENSAL'] - df.iloc[0]['DEMSAL']
-    net_production['HN'] = df.iloc[0]['GENHON'] - df.iloc[0]['DEMHON']
-    net_production['NI'] = df.iloc[0]['GENNIC'] - df.iloc[0]['DEMNIC']
-    net_production['CR'] = df.iloc[0]['GENCRI'] - df.iloc[0]['DEMCRI']
-    net_production['PA'] = df.iloc[0]['GENPAN'] - df.iloc[0]['DEMPAN']
-    net_production['MX'] = df.iloc[0]['TOTALGEN'] - df.iloc[0]['TOTALDEM']
-
-    return net_production
-
-
-def flow_logic(net_production, interconnections):
-    """
-    Calculates flow direction for each interconnection using network flow and
-    simultaneous equations.
-    Returns a dictionary.
-    """
-
-    # Each country is modeled as a node with flows going either in or out of it.
-    # Importing is given a negative flow while exporting is positive.
-
-    PA = {'CR': 0}
-    CR = {'PA': 0, 'NI': 0}
-    NI = {'CR': 0, 'HN': 0}
-    HN = {'GT': 0, 'SV': 0, 'NI': 0}
-    SV = {'GT': 0, 'HN': 0}  # TODO: SV is assigned to but never used
-    GT = {'MX': 0, 'HN': 0, 'SV': 0}
-
-    def plusminus(value):
-        """
-        Takes a number and check its sign.
-        Returns 1 if positive, -1 if negative and zero if zero.
-        """
-
-        if value > 0:
-            newvalue = 1
-        elif value < 0:
-            newvalue = -1
+    if exchange in ['GT->SV', 'GT->HN', 'HN->SV', 'CR->NI']:
+        if logic == 0:
+            return -1
         else:
-            newvalue = 0
-
-        return newvalue
-
-    def flipsign(value):
-        """
-        Changes the sign of any number given apart from zero.
-        1 -> -1
-        -1 -> 1
-        """
-
-        newvalue = (-1) * value
-
-        return newvalue
-
-    flows = {'HN->NI': 0.0,
-             'CR->NI': 0.0,
-             'CR->PA': 0.0,
-             'GT->HN': 0.0,
-             'GT->MX': 0.0,
-             'GT->SV': 0.0,
-             'HN->SV': 0.0}
-
-    # First we determine whether Mexico is importing or exporting using totals for the SIEPAC system.
-
-    if net_production['MX'] < 0:
-        # exporting
-        GT['MX'] = -1
+            return 1
     else:
-        GT['MX'] = 1
-
-    # We then find the direction of the PA by exploiting the fact that it only has one interconnection.
-
-    if net_production['PA'] > 0:
-        # PA can only export to CR
-        PA['CR'] = 1
-        CR['PA'] = -1
-    else:
-        # PA importing from CR
-        PA['CR'] = -1
-        CR['PA'] = 1
-
-    # Next we can find CR and NI flows using their net productions and process of elimination.
-
-    PAN = interconnections['CR->PA'] * CR['PA']
-
-    CR['NI'] = plusminus((net_production['CR'] - PAN) / interconnections['CR->NI'])
-    NI['CR'] = flipsign(CR['NI'])
-    NIC = interconnections['CR->NI'] * NI['CR']
-
-    NI['HN'] = plusminus((net_production['NI'] - NIC) / interconnections['HN->NI'])
-    HN['NI'] = flipsign(NI['HN'])
-
-    # Now we use 3 simultaneous equations to find the remaining flows.  We can use the fact that
-    # several flows are already known to our advantage.
-
-    # a = interconnections['GT->SV']
-    # b = interconnections['HN->SV']
-    # c = interconnections['GT->HN']
-    # MX = interconnections['GT->MX']*GT['MX']
-    # HON = interconnections['HN->NI']*HN['NI']
-    #
-    # eqs = np.array([[a, b, 0], [a, 0, c], [0, b, c]])
-    # res = np.array([net_production['SV'], net_production['GT']-MX, net_production['HN']-HON])
-    #
-    # solution = np.linalg.solve(eqs, res)
-    #
-    # #Factor to account for transmission losses.
-    # GT['SV'] = plusminus(solution[0]+0.5)
-    #
-    # SV['GT'] = flipsign(GT['SV'])
-    # SV['HN'] = plusminus(solution[1])
-    # HN['SV'] = flipsign(SV['HN'])
-    # GT['HN'] = plusminus(solution[2])
-    # HN['GT'] = flipsign(GT['HN'])
-
-    # Flows commented out are disabled until the maths behind determining their direction can be proved satisfactorily.
-    flows['HN->NI'] = HN['NI']
-    flows['CR->NI'] = CR['NI']
-    flows['CR->PA'] = CR['PA']
-    # flows['GT->HN'] = GT['HN']
-    flows['GT->MX'] = GT['MX']
-    # flows['GT->SV'] = GT['SV']
-    # flows['HN->SV'] = SV['HN']
-
-    return flows
-
-
-def net_flow(interconnections, flows):
-    """
-    Combines interconnection values with flow directions.
-    Returns a dictionary.
-    """
-
-    netflow = {k: interconnections[k] * flows[k] for k in interconnections}
-
-    return netflow
+        if logic == 1:
+            return -1
+        else:
+            return 1
 
 
 def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, logger=None):
@@ -197,36 +67,36 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
     if target_datetime:
         raise NotImplementedError('This parser is not yet able to parse past dates')
 
-    getdata = read_data()
-    connect = connections(getdata)
-    nt = net(getdata)
-    fl = flow_logic(nt, connect)
-    netflow = net_flow(connect, fl)
-
-    exchange = {}
-    dt = arrow.now('UTC-6').floor('minute')
-    zones = '->'.join(sorted([zone_key1, zone_key2]))
+    sorted_zones = '->'.join(sorted([zone_key1, zone_key2]))
 
-    if zones in netflow:
-        exchange['netFlow'] = netflow[zones]
-    else:
+    if sorted_zones not in CSV_MAPPING.keys():
         raise NotImplementedError('This exchange is not implemented.')
 
-    exchange.update(sortedZoneKeys=zones,
-                    datetime=dt.datetime,
-                    source='enteoperador.org')
+    raw_data = pd.read_csv(DATA_URL, index_col=False)
+    flow = extract_exchange(raw_data, sorted_zones)
+    dt = arrow.now('UTC-6').floor('minute')
+
+    exchange = {'sortedZoneKeys': sorted_zones,
+                'datetime': dt.datetime,
+                'netFlow': flow,
+                'source': 'enteoperador.org'}
 
     return exchange
 
 
 if __name__ == '__main__':
     """Main method, never used by the Electricity Map backend, but handy for testing."""
-
     print('fetch_exchange(CR, PA) ->')
     print(fetch_exchange('CR', 'PA'))
     print('fetch_exchange(CR, NI) ->')
     print(fetch_exchange('CR', 'NI'))
-    print('fetch_exchange(HN, NI) ->')
-    print(fetch_exchange('HN', 'NI'))
+    # print('fetch_exchange(HN, NI) ->')
+    # print(fetch_exchange('HN', 'NI'))
     print('fetch_exchange(GT, MX) ->')
     print(fetch_exchange('GT', 'MX'))
+    print('fetch_exchange(GT, SV) ->')
+    print(fetch_exchange('GT', 'SV'))
+    print('fetch_exchange(GT, HN) ->')
+    print(fetch_exchange('GT', 'HN'))
+    print('fetch_exchange(HN, SV) ->')
+    print(fetch_exchange('HN', 'SV'))


Test patch: