Repository: electricitymaps/electricitymaps-contrib

Issue:
Uruguay data too good to be true/shown?
Aren't we missing dispatchable power production? Hydro, thermal, or imports?
100% intermittent renewable consumption-mix is not possible: We should probably consider this data as incorrect until we find better data source

![image](https://user-images.githubusercontent.com/22095643/38616976-1030df5e-3d96-11e8-9ee3-90b86bac4da0.png)




Discussion:
@brunolajoie exchanges with Argentina are missing.
See comments from #565 
https://github.com/tmrowco/electricitymap/issues/565#issuecomment-376188937
@brunolajoie and you are right, something else is missing:
![image](https://user-images.githubusercontent.com/25743609/38618321-8f6f32d6-3d99-11e8-96b6-5099a7e570d5.png)

the red part "compras SG" which translates to "shopping SG" ^^ so maybe that is the energy from the Salto Grande dam? Or purchased from Argentina?
_ _ _ _
EDIT: I think the red part is the share of the generation of SG that is remaining in UY and doesn't flow to AR.
Trying to interpret this scheme from the AR website (art classes from schooltime finally pay off 🎉 ):
SGA (Salto Grande Argentina) and SGU (Uruguay) hydro were generating 682+794 MW last hour.
So 794 MW are generated "in Uruguay" but a large share (~300 MW) of this is leaving the scheme towards Argentina via the direct interconnector between SGU and SGA. Another huge part (~500 MW) is leaving via SJA (San Javier, UY) to CEL (in AR). What remains in UY are these 53+54 MW, but this is unimportant, because it is not imported, but an country-internal flow. Does that make sense?
[LINK](http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Tensiones%20y%20Flujos%20de%20Potencia&11/04/2018%2009:00)
![image](https://user-images.githubusercontent.com/25743609/38618958-56c2351c-3d9b-11e8-9104-902c380e342d.png)

We are missing the hydro generation of 794 MW in that hour at SGU. 
It's not the imports that are missing, but the hydro generation of Salto Grande in Uruguay.
The exports to Argentina are HUUUUGE. 311 + 506 MW from that screenshot ;) 
@systemcatch regarding #565 this is very important for the interconnectors of AR <-> UY.
I will leave this info there as well!

What is my proposed solution? Grab hydro generation for Salto Grande from Cammesa/Argentina and not from UTE/Uruguay and use both values of these horizontal lines for exports and imports.

Resolving patch:
diff --git a/parsers/AR.py b/parsers/AR.py
--- a/parsers/AR.py
+++ b/parsers/AR.py
@@ -96,7 +96,7 @@
     'BRKETG01': 'gas',
     'BRKETG02': 'gas',
     'BRKETG03': 'gas',
-    'BRKETG04': 'gas',    
+    'BRKETG04': 'gas',
     'BROWTG01': 'gas',
     'BROWTG02': 'gas',
     'BSASTG01': 'gas',
@@ -519,6 +519,12 @@
 
 cammesa_url = 'http://portalweb.cammesa.com/default.aspx'
 
+tie_mapping = {'CL-SING': "position:absolute; top:349; left:585",
+               'PY': "position:absolute; top:67; left:649",
+               'UY_1': "position:absolute; top:203; left:533",
+               'UY_2': "position:absolute; top:226; left:515"
+               }
+
 
 def webparser(req):
     """Takes content from webpage and returns all text as a list of strings"""
@@ -781,6 +787,106 @@ def fetch_production(zone_key='AR', session=None, target_datetime=None, logger=N
     return production_mix
 
 
+def direction_finder(direction, exchange):
+    """
+    Uses the 'src' attribute of an "img" tag to find the
+    direction of flow. In the data source small arrow images
+    are used to show flow direction.
+    """
+
+    if direction == "/uflujpot.nsf/f90.gif":
+        # flow from Argentina
+        return 1
+    elif direction == "/uflujpot.nsf/f270.gif":
+        # flow to Argentina
+        return -1
+    else:
+        raise ValueError('Flow direction for {} cannot be determined, got {}'.format(exchange, direction))
+
+
+def tie_finder(exchange_url, exchange, session):
+    """
+    Finds tie data using div tag style attribute.
+    Returns a float.
+    """
+
+    req = session.get(exchange_url)
+    soup = BeautifulSoup(req.text, 'html.parser')
+
+    tie = soup.find("div", style = tie_mapping[exchange])
+    flow = float(tie.text)
+    direction_tag = tie.find_next("img")
+    direction = direction_finder(direction_tag['src'], exchange)
+
+    netflow = flow*direction
+
+    return netflow
+
+
+def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, logger=None):
+    """Requests the last known power exchange (in MW) between two zones
+
+    Arguments:
+    zone_key1, zone_key2: specifies which exchange to get
+    session: requests session passed in order to re-use an existing session,
+      not used here due to difficulty providing it to pandas
+    target_datetime: the datetime for which we want production data. If not provided, we should
+      default it to now. The provided target_datetime is timezone-aware in UTC.
+    logger: an instance of a `logging.Logger`; all raised exceptions are also logged automatically
+
+    Return:
+    A list of dictionaries in the form:
+    {
+      'sortedZoneKeys': 'DK->NO',
+      'datetime': '2017-01-01T00:00:00Z',
+      'netFlow': 0.0,
+      'source': 'mysource.com'
+    }
+    where net flow is from DK into NO
+    """
+
+    # Only hourly data is available.
+    if target_datetime:
+        lookup_time = arrow.get(target_datetime).floor('hour').format('DD/MM/YYYY HH:mm')
+    else:
+        current_time = arrow.now('UTC-3')
+        if current_time.minute < 30:
+            # Data for current hour seems to be available after 30mins.
+            current_time = current_time.shift(hours=-1)
+        lookup_time = current_time.floor('hour').format('DD/MM/YYYY HH:mm')
+
+    sortedcodes = '->'.join(sorted([zone_key1, zone_key2]))
+
+    if sortedcodes == 'AR->CL-SING':
+        base_url = 'http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Unifilar de NOA&'
+    else:
+        base_url = 'http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Tensiones y Flujos de Potencia&'
+
+    exchange_url = base_url + lookup_time
+
+    s = session or requests.Session()
+
+    if sortedcodes == 'AR->UY':
+        first_tie = tie_finder(exchange_url, 'UY_1', s)
+        second_tie = tie_finder(exchange_url, 'UY_2', s)
+        flow = first_tie + second_tie
+    elif sortedcodes == 'AR->PY':
+        flow = tie_finder(exchange_url, 'PY', s)
+    elif sortedcodes == 'AR->CL-SING':
+        flow = tie_finder(exchange_url, 'CL-SING', s)
+    else:
+        raise NotImplementedError('This exchange is not currently implemented')
+
+    exchange = {
+      'sortedZoneKeys': sortedcodes,
+      'datetime': arrow.now('UTC-3').datetime,
+      'netFlow': flow,
+      'source': 'cammesa.com'
+    }
+
+    return exchange
+
+
 if __name__ == '__main__':
     """Main method, never used by the Electricity Map backend, but handy for testing."""
 
@@ -788,3 +894,9 @@ def fetch_production(zone_key='AR', session=None, target_datetime=None, logger=N
     print(fetch_production())
     print('fetch_price() ->')
     print(fetch_price())
+    print('fetch_exchange(AR, PY) ->')
+    print(fetch_exchange('AR', 'PY'))
+    print('fetch_exchange(AR, UY) ->')
+    print(fetch_exchange('AR', 'UY'))
+    print('fetch_exchange(AR, CL-SING) ->')
+    print(fetch_exchange('AR', 'CL-SING'))
diff --git a/parsers/UY.py b/parsers/UY.py
--- a/parsers/UY.py
+++ b/parsers/UY.py
@@ -19,6 +19,31 @@
 }
 INV_MAP_GENERATION = dict([(v, k) for (k, v) in MAP_GENERATION.items()])
 
+SALTO_GRANDE_URL = 'http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Tensiones y Flujos de Potencia&'
+
+
+def get_salto_grande(session):
+    """
+    Finds the current generation from the Salto Grande Dam that is
+    allocated to Uruguay.
+    """
+
+    current_time = arrow.now('UTC-3')
+    if current_time.minute < 30:
+        # Data for current hour seems to be available after 30mins.
+        current_time = current_time.shift(hours=-1)
+    lookup_time = current_time.floor('hour').format('DD/MM/YYYY HH:mm')
+
+    s = session or requests.Session()
+    url = SALTO_GRANDE_URL + lookup_time
+    response = s.get(url)
+    soup = BeautifulSoup(response.text, 'html.parser')
+
+    tie = soup.find("div", style = "position:absolute; top:143; left:597")
+    generation = float(tie.text)
+
+    return generation
+
 
 def parse_page(session):
     r = session or requests.session()
@@ -63,11 +88,9 @@ def parse_page(session):
 
         obj[k] = v
 
-    # Note that the Salto Grande is counted as hydro, whereas it really is a hydro *import*
-    # However, because it is coming directly from the hydro dam, we can count it as local production
-    # and not as an import. This should be verified though.
-    # If this assumption is not correct, then the following line should be uncommented.
-    # obj['Hidráulica'] -= max(0, obj['Interconexión Salto Grande'])
+    # https://github.com/tmrowco/electricitymap/issues/1325#issuecomment-380453296
+    salto_grande = get_salto_grande(session)
+    obj['Hidráulica'] = obj.get('Hidráulica', 0.0) + salto_grande
 
     return obj
 
@@ -75,7 +98,7 @@ def parse_page(session):
 def fetch_production(zone_key='UY', session=None, target_datetime=None, logger=None):
     if target_datetime:
         raise NotImplementedError('This parser is not yet able to parse past dates')
-    
+
     obj = parse_page(session)
 
     data = {
@@ -106,7 +129,7 @@ def fetch_exchange(zone_key1='UY', zone_key2='BR-S', session=None, target_dateti
     """
     if target_datetime:
         raise NotImplementedError('This parser is not yet able to parse past dates')
-    
+
     # set comparison
     if {zone_key1, zone_key2} != {'UY', 'BR'}:
         return None
@@ -126,5 +149,7 @@ def fetch_exchange(zone_key1='UY', zone_key2='BR-S', session=None, target_dateti
 
 
 if __name__ == '__main__':
+    print('fetch_production() ->')
     print(fetch_production())
+    print('fetch_exchange(UY, BR) ->')
     print(fetch_exchange('UY', 'BR'))


Test patch: