Repository: electricitymaps/electricitymaps-contrib

Issue:
Revise and improve AR.py plant mapping
Data for categories ["thermal"](http://portalweb.cammesa.com/MEMNet1/Pages/Informes%20por%20Categor%C3%ADa/Operativos/DespachoGenTermica_Minimal.aspx) and ["hydro"](http://portalweb.cammesa.com/memnet1/Pages/Informes%20por%20Categor%C3%ADa/Operativos/DespachoGenHidra_Minimal.aspx) generation from Cammesa is used for the map.
Some recently comissioned power plants were causing a warning every once in a while, because they were not yet included in the type mapping. This could be fixed by improving the parser a bit. 

I just copy and edit the rest of my comment from #1284:
_ _ _ _

1. There is a clear pattern behind the power plant IDs: In general, everything that ends with "TG" before the numeric part is a gas turbine and should be running on natural gas as main fuel (e.g. ABCD**TG**01)
Actually, the plant mapping in the AR.py could be entirely dismissed, when only the the fourth and fifth letter of the plant IDs will be considered.

2. The mapping could be refined and updated by using the first column on
http://portalweb.cammesa.com/memnet1/revistas/estacional/base_gen.html for all of the existing power plants. 

![image](https://user-images.githubusercontent.com/25743609/38309401-4b0e623a-381a-11e8-85da-af7530e82a87.png)

Could that page also be used to map the reporting plants by their IDs automatically? (see mapping below)
- Here is the mapping of that first column to the fuel types (from what I could relate to the power plant IDs):
* TG is gas (turbina de gas)
* CC is gas (combined cycle)
* AG is gas (no idea what the A means, but the web leads me to CCGT plants)
* TV is mostly gas in the recent mapping - maybe this category needs a seperate review:
  * In the recent mapping, 5 of the steam turbines TV (turbina de vapor) are running on coal (~170 MW), which seems easy enough to consider, the rest of TV should be using gas, too. Maybe some biomass is included here as well.
* DI is oil (diesel)
* EO is wind (eolico)
* HI is hydro (hidro)
* FV is solar (fotovoltaica)
* BG is biomass (biogeneradora)
* BM is biomass (biomassa)
* NU is nuclear
* BRS and MH have a very tiny amount of MWs, I suggest mapping them to "unknown" together with any other abbreviation they might come up with in the future.

This would eliminate/limit this issue that pops up every once in a while when a new unit (or previously unused unit) suddenly enters operation.

I went through the mapping in the parser and found some errors as well, like "EO" wind power plants and "FV" solar power plants recently being mapped to "hydro". Probably because they are included in the page with hydro generation data. There are only 5 of these, that need to be changed, so that has no huge impact.


Discussion:
I'd rather not make assumptions about the power plants based on 2 letters tbh. Better we just update them as necessary which has worked ok so far.

Good spot with the EO and FV plants, it's weird they're in the hydro section but possibly that's more of a renewable section now anyway. I probably assumed everything was hydro there when I built the parser.
@systemcatch and I had a brief chat on this. Mapping the entire power plant list from http://portalweb.cammesa.com/memnet1/revistas/estacional/base_gen.html is the easiest solution.
It also includes units to be commissioned in the near future, so the warnings will be avoided to some extent.
Awesome that sounds like a good plan
@alixunderplatz are you still able to do this one?
@systemcatch nope, not much spare time atm, sorry 😢 I even forgot I had self-assigned this one back then^^

Resolving patch:
diff --git a/parsers/AR.py b/parsers/AR.py
--- a/parsers/AR.py
+++ b/parsers/AR.py
@@ -3,7 +3,7 @@
 import itertools
 import re
 import string
-
+import logging
 import arrow
 import requests
 from bs4 import BeautifulSoup
@@ -36,6 +36,7 @@
     'ACAJTG05': 'gas',
     'ACAJTG06': 'gas',
     'ACAJTV07': 'gas',
+    'ACHIEO': 'wind',
     'ADTOHI': 'hydro',
     'AESPTG01': 'gas',
     'AESPTG02': 'gas',
@@ -56,8 +57,10 @@
     'ANCHDI03': 'oil',
     'ANCHDI04': 'oil',
     'APARTV01': 'gas',
-    'ARA2EO': 'hydro',
-    'ARAUEO': 'hydro',
+    'ARA2EO': 'wind',
+    'ARA3EO': 'wind',
+    'ARAUEO': 'wind',
+    'ARC2EO': 'wind',
     'ARGETG01': 'gas',
     'ARISDI01': 'oil',
     'ARMATG01': 'gas',
@@ -111,6 +114,7 @@
     'CAIMDI03': 'oil',
     'CAIMDI04': 'oil',
     'CAIMDI05': 'oil',
+    'CALOFV': 'solar',
     'CARLDI01': 'oil',
     'CARRHI': 'hydro',
     'CASSHI': 'hydro',
@@ -134,8 +138,10 @@
     'CHARDI01': 'oil',
     'CHARDI02': 'oil',
     'CHEPDI01': 'oil',
+    'CHI1FV': 'solar',
     'CHILDI01': 'oil',
     'CHLEDI01': 'oil',
+    'CHNOEO': 'wind',
     'CHOCHI': 'hydro',
     'CIPODI01': 'oil',
     'CIPOHI': 'hydro',
@@ -144,6 +150,7 @@
     'CONDHI': 'hydro',
     'COROHI': 'hydro',
     'CORRDI01': 'gas',
+    'CORTEO': 'wind',
     'COSMDI11': 'oil',
     'COSTTG08': 'gas',
     'COSTTG09': 'gas',
@@ -156,12 +163,15 @@
     'COSTTV10': 'gas',
     'CPIEHI': 'hydro',
     'CSARDI01': 'oil',
+    'CUMBFV': 'solar',
     'CUMODI01': 'gas',
     'CURUTG01': 'gas',
     'CURUTG02': 'gas',
+    'CVIEHI': 'hydro',
     'DFUNDI01': 'oil',
+    'DFUNTG01': 'gas',
     'DFUNTG02': 'gas',
-    'DIADEO': 'hydro',
+    'DIADEO': 'wind',
     'DIQUTG02': 'gas',
     'DIQUTG03': 'gas',
     'DSUDTG07': 'gas',
@@ -176,6 +186,7 @@
     'EMBANUCL': 'nuclear',
     'ESCAHI': 'hydro',
     'ESQDDI01': 'oil',
+    'ETIGHI': 'hydro',
     'EZEITG01': 'gas',
     'EZEITG02': 'gas',
     'EZEITG03': 'gas',
@@ -184,6 +195,7 @@
     'FRIATG01': 'gas',
     'FSIMHI': 'hydro',
     'FUTAHI': 'hydro',
+    'GARAEO': 'wind',
     'GBELTG01': 'gas',
     'GBELTG02': 'gas',
     'GBELTV01': 'gas',
@@ -197,10 +209,13 @@
     'GUEMTV11': 'gas',
     'GUEMTV12': 'gas',
     'GUEMTV13': 'gas',
-    'HON1FV': 'hydro',
+    'HON1FV': 'solar',
+    'HON2FV': 'solar',
     'HRENDI01': 'oil',
     'HUMADI01': 'oil',
     'HUEMDI01': 'gas',
+    #'IMACE1UR': 'unknown', likely in Uruguay
+    #'IMACE2UR': 'unknown', likely in Uruguay
     'INDETG01': 'gas',
     'INDETG02': 'gas',
     'INDETG03': 'gas',
@@ -216,6 +231,7 @@
     'LBLADI01': 'oil',
     'LCA2TG01': 'gas',
     'LCAMTG01': 'gas',
+    'LCASEO': 'wind',
     'LDCUHI': 'hydro',
     'LDCUTG22': 'gas',
     'LDCUTG23': 'gas',
@@ -241,7 +257,8 @@
     'LMADHI': 'hydro',
     'LMO1HI': 'hydro',
     'LMO2HI': 'hydro',
-    'LOM1EO': 'hydro',
+    'LOM1EO': 'wind',
+    'LOM4EO': 'wind',
     'LOBODI01': 'oil',
     'LPALDI01': 'oil',
     'LPAZDI01': 'oil',
@@ -257,9 +274,16 @@
     'LRIPDI01': 'oil',
     'LRISDI01': 'oil',
     'LROBDI01': 'oil',
+    'LUJBTG01': 'gas',
+    'LUJBTG02': 'gas',
     'LVARDI01': 'oil',
     'LVINHI': 'hydro',
     'MAGDDI01': 'oil',
+    'MANAEO': 'wind',
+    'MAT3TG01': 'gas',
+    'MAT3TG02': 'gas',
+    'MAT3TG03': 'gas',
+    'MAT3TG04': 'gas',
     'MATETG01': 'gas',
     'MATETG02': 'gas',
     'MATETG03': 'gas',
@@ -295,7 +319,7 @@
     'MMARTG06': 'gas',
     'MMARTG07': 'gas',
     'MSEVTG01': 'gas',
-    'NECOEO': 'hydro',
+    'NECOEO': 'wind',
     'NECOTV01': 'gas',
     'NECOTV02': 'gas',
     'NECOTV03': 'gas',
@@ -304,6 +328,7 @@
     'NIH1HI': 'hydro',
     'NIH4HI': 'hydro',
     'NOMODI01': 'gas',
+    'NONOFV': 'solar',
     'NPOMDI01': 'gas',
     'NPUETV05': 'gas',
     'NPUETV06': 'gas',
@@ -360,6 +385,7 @@
     'PINATG10': 'gas',
     'PIQIDI01': 'oil',
     'PIRADI01': 'oil',
+    'PMA1EO': 'wind',
     'PMORHI': 'hydro',
     'PNEGHI': 'hydro',
     'PNUETV07': 'gas',
@@ -381,13 +407,16 @@
     'PZUEDI01': 'oil',
     'QULLHI': 'hydro',
     'RAFADI01': 'oil',
-    'RAW1EO': 'hydro',
-    'RAW2EO': 'hydro',
+    'RAW1EO': 'wind',
+    'RAW2EO': 'wind',
+    'RAW3EO': 'wind',
     'RCEPDI01': 'oil',
     'RCUATG02': 'gas',
     'REALDI01': 'oil',
+    'RENOTG01': 'gas',
     'REOLHI': 'hydro',
     'RESCDI01': 'oil',
+    'RESCHI': 'hydro',
     'RGDEHB': 'hydro',
     'RHONHI': 'hydro',
     'RICADI01': 'oil',
@@ -408,6 +437,7 @@
     'SARCTG21': 'gas',
     'SARCTG22': 'gas',
     'SARCTG23': 'gas',
+    'SAUJFV': 'solar',
     'SCHADI01': 'oil',
     'SCTPDI01': 'oil',
     'SERTTG01': 'gas',
@@ -417,10 +447,11 @@
     'SGDEHIAR': 'hydro',
     'SGUIHI': 'hydro',
     'SHELTG01': 'gas',
-    'SJUAFV': 'hydro',
+    'SJUAFV': 'solar',
     'SLTODI01': 'oil',
     'SMANDI01': 'oil',
     'SMARDI01': 'oil',
+    'SMARHI': 'hydro',
     'SMIGDI01': 'oil',
     'SMTUTG01': 'gas',
     'SMTUTG02': 'gas',
@@ -451,12 +482,17 @@
     'TIMBTG02': 'gas',
     'TIMBTV01': 'gas',
     'TINODI01': 'oil',
-    'TORDEO': 'hydro',
+    'TORDEO': 'wind',
     'TUCUTG01': 'gas',
     'TUCUTG02': 'gas',
     'TUCUTV01': 'gas',
     'TUNAHI': 'hydro',
+    #'UTEIHICO': 'unknown', Constitucion Dam in Uruguay
+    #'UTEIHISG': 'unknown', Salto Grande Dam in Uruguay
+    'ULL3FV': 'solar',
     'ULLUHI': 'hydro',
+    'ULN1FV': 'solar',
+    'ULN2FV': 'solar',
     'VANGDI01': 'oil',
     'VGADDI01': 'oil',
     'VGEPDI01': 'oil',
@@ -465,6 +501,7 @@
     'VGESTG16': 'gas',
     'VGESTG18': 'gas',
     'VIALDI01': 'oil',
+    'VLONEO': 'wind',
     'VMA2TG01': 'gas',
     'VMA2TG02': 'gas',
     'VMA2TG03': 'gas',
@@ -538,7 +575,7 @@ def webparser(req):
     return data_table
 
 
-def fetch_price(zone_key='AR', session=None, target_datetime=None, logger=None):
+def fetch_price(zone_key='AR', session=None, target_datetime=None, logger=logging.getLogger(__name__)):
     """
     Requests the last known power price of a given country
     Arguments:
@@ -617,6 +654,17 @@ def dataformat(junk):
     return formatted
 
 
+def generation_finder(data, gen_type):
+    """Finds all genration matching requested type in a list.
+    Sums together and returns a float.
+    """
+
+    find_generation = [i + 2 for i, x in enumerate(data) if x == gen_type]
+    generation_total = sum([data[i] for i in find_generation])
+
+    return generation_total
+
+
 def get_thermal(session, logger):
     """
     Requests thermal generation data then parses and sorts by type.  Nuclear is included.
@@ -657,33 +705,24 @@ def get_thermal(session, logger):
     formatted_data = dataformat(data)
     mapped_data = [power_plant_type.get(x, x) for x in formatted_data]
 
-    for item in mapped_data:
+    for idx, item in enumerate(mapped_data):
         try:
             # avoids including titles and headings
-            if all((item.isupper(), not item.isalpha(), ' ' not in item)):
+            if all((item.isupper(), item.isalnum(), item != 'MERCADO')):
                 logger.warning(
                     '{} is missing from the AR plant mapping!'.format(item),
                     extra={'key': 'AR'})
+                mapped_data[idx] = 'unknown'
         except AttributeError:
             # not a string....
             continue
 
-    find_totals = [i + 1 for i, x in enumerate(mapped_data) if x == 'Totales ']
-    thermal_generation = sum([mapped_data[i] for i in find_totals])
-
-    find_nuclear = [i + 2 for i, x in enumerate(mapped_data) if x == 'nuclear']
-    nuclear_generation = sum([mapped_data[i] for i in find_nuclear])
-    find_oil = [i + 2 for i, x in enumerate(mapped_data) if x == 'oil']
-    oil_generation = sum([mapped_data[i] for i in find_oil])
-    find_coal = [i + 2 for i, x in enumerate(mapped_data) if x == 'coal']
-    coal_generation = sum([mapped_data[i] for i in find_coal])
-    find_biomass = [i + 2 for i, x in enumerate(mapped_data) if x == 'biomass']
-    biomass_generation = sum([mapped_data[i] for i in find_biomass])
-    find_gas = [i + 2 for i, x in enumerate(mapped_data) if x == 'gas']
-    gas_generation = sum([mapped_data[i] for i in find_gas])
-
-    unknown_generation = (thermal_generation - nuclear_generation - gas_generation
-                          - oil_generation - coal_generation - biomass_generation)
+    nuclear_generation = generation_finder(mapped_data, 'nuclear')
+    oil_generation = generation_finder(mapped_data, 'oil')
+    coal_generation = generation_finder(mapped_data, 'coal')
+    biomass_generation = generation_finder(mapped_data, 'biomass')
+    gas_generation = generation_finder(mapped_data, 'gas')
+    unknown_generation = generation_finder(mapped_data, 'unknown')
 
     if unknown_generation < 0.0:
         unknown_generation = 0.0
@@ -693,12 +732,13 @@ def get_thermal(session, logger):
             'coal': coal_generation,
             'unknown': unknown_generation,
             'oil': oil_generation,
-            'biomass': biomass_generation
-            }
+            'biomass': biomass_generation}
 
 
-def get_hydro(session=None):
-    """Requests hydro generation data then parses, returns a dictionary."""
+def get_hydro_and_renewables(session, logger):
+    """Requests hydro generation data then parses into a usable format.
+    There's sometimes solar and wind plants included in the data.
+    Returns a dictionary."""
 
     s = session or requests.Session()
     r = s.get(hurl)
@@ -723,13 +763,32 @@ def get_hydro(session=None):
 
     data = list(itertools.chain.from_iterable(full_table))
     formatted_data = dataformat(data)
-    find_hydro = [i + 1 for i, x in enumerate(formatted_data) if x == 'Totales ']
-    total_hydro_generation = sum([formatted_data[i] for i in find_hydro])
+    mapped_data = [power_plant_type.get(x, x) for x in formatted_data]
+
+    for idx, item in enumerate(mapped_data):
+        try:
+            # avoids including titles and headings
+            if all((item.isupper(), item.isalnum(), item != 'MERCADO')):
+                logger.warning(
+                    '{} is missing from the AR plant mapping!'.format(item),
+                    extra={'key': 'AR'})
+                mapped_data[idx] = 'unknown'
+        except AttributeError:
+            # not a string....
+            continue
+
+    hydro_generation = generation_finder(mapped_data, 'hydro')
+    solar_generation = generation_finder(mapped_data, 'solar')
+    wind_generation = generation_finder(mapped_data, 'wind')
+    unknown_generation = generation_finder(mapped_data, 'unknown')
 
-    return {'hydro': total_hydro_generation}
+    return {'hydro': hydro_generation,
+            'wind': wind_generation,
+            'solar': solar_generation,
+            'unknown': unknown_generation}
 
 
-def fetch_production(zone_key='AR', session=None, target_datetime=None, logger=None):
+def fetch_production(zone_key='AR', session=None, target_datetime=None, logger=logging.getLogger(__name__)):
     """
     Requests the last known production mix (in MW) of a given country
     Arguments:
@@ -764,22 +823,16 @@ def fetch_production(zone_key='AR', session=None, target_datetime=None, logger=N
 
     gdt = get_datetime(session=None)
     thermal = get_thermal(session, logger)
-    hydro = get_hydro(session=None)
+    hydro = get_hydro_and_renewables(session, logger)
+
+    unknown = thermal.pop('unknown') + hydro.pop('unknown')
+    production = {**hydro, **thermal}
+    production['unknown'] = unknown
+
     production_mix = {
         'zoneKey': zone_key,
         'datetime': gdt['datetime'],
-        'production': {
-            'biomass': thermal.get('biomass', 0.0),
-            'coal': thermal.get('coal', 0.0),
-            'gas': thermal.get('gas', 0.0),
-            'hydro': hydro.get('hydro', 0.0),
-            'nuclear': thermal.get('nuclear', 0.0),
-            'oil': thermal.get('oil', 0.0),
-            'solar': None,
-            'wind': None,
-            'geothermal': 0.0,
-            'unknown': thermal.get('unknown', 0.0)
-        },
+        'production': production,
         'storage': {
             'hydro': None,
         },


Test patch: