Repository: electricitymaps/electricitymaps-contrib

Issue:
IN-GJ (Gujarat) parser needs improvement
I was about to add capacity data to **IN-GJ** and looked up the data source from the parser for conventional generation.
https://www.sldcguj.com/RealTimeData/RealTimeDemand.php
It also has capacity data for the units of the power plants.

1. While cross checking with the map, I found it highly confusing that "Gujarat Total Conventional Generation" on that site (about 8500 MW) was above the total value for all combined types in the origin graph.

![image](https://user-images.githubusercontent.com/25743609/38413298-8d85219a-398b-11e8-9d29-188c72442993.png)

2. GMR is mapped to wind, but should be something thermal PP as well (I suggest to "unknown" - the wind numbers are in a seperate figure).

3. CGPL is mapped to coal, with a current generation of 2530 MW, which is totally missing on the map.
I assume the two spaces "  " behind "CGPL  " may cause this trouble.
This is a huge amount that is missing in the "coal" category and would fill up a large part of the gap.

4. The "unknown" category could be filled with what is mapped as "thermal" in the parser. That figure is definitely missing on the map. (~2800 MW)

5. Here is a screenshot of the recent mapping. I colored the fields according to their recent mapping. I also edited the fields with "123 + 456" to be the sum of both (generating units)

![image](https://user-images.githubusercontent.com/25743609/38414879-0ac87d1a-3990-11e8-9521-d519b4ef59cd.png)


Discussion:
I had confused something in the previous comment and fixed it ;-)
Now the issue is described correctly
Some more infos on the grey "unknown" thermal types in the image above.
This site was very in helpful: https://en.wikipedia.org/wiki/Category:Coal-fired_power_stations_in_Gujarat

[Ukai](https://en.wikipedia.org/wiki/Ukai_Thermal_Power_Station): coal
[Wanakabori](https://en.wikipedia.org/wiki/Wanakbori_Thermal_Power_Station): coal
[Gandhinagar](https://en.wikipedia.org/wiki/Gandhinagar_Thermal_Power_Station): coal
[Sikka](https://en.wikipedia.org/wiki/Sikka_Thermal_Power_Station): coal
[KLTPS (= Kutch LTPS)](https://en.wikipedia.org/wiki/Kutch_Thermal_Power_Station): coal (for future reference: lignite)
[SLPP (=Surat LPP)](https://en.wikipedia.org/wiki/Surat_Thermal_Power_Station): coal (lignite)
[Akrimota](https://en.wikipedia.org/wiki/Akrimota_Thermal_Power_Station): coal (lignite)

[JHANOR](https://en.wikipedia.org/wiki/Jhanor-Gandhar_Thermal_Power_Station): gas

I assume this strange "GMR" with 685 MW installed power is contracted and located outside of gujarat in the state Chhattisgarh. There is[ a coal power plant with 2x 685 MW](http://gmrgroup.in/energy-gmr-chhattisgarh-power-project.aspx), which might have sold a part of its output to Gujarat, according to [this news article: "State electricity department officials told FE that GMR Chattisgarh offered to sell power to the state under the ‘flexible utilisation of coal’ scheme. "](http://www.financialexpress.com/industry/gujarats-tender-gets-dismal-response/893920/)
So GMR can be dismissed, as it is most likely not located in GJ, and removed from "wind".

Everything is clear now, so the [parser](https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_GJ.py) can be updated / improved.
I can't test locally, so can someone try and see if the updated mapping below for [IN_GJ.py](https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_GJ.py) still works? 😸
Would be nice to check if every single value is really grabbed, because sometimes one or even two " " spaces are behind the names of the power plants. Plenty of coal should be returned now.
I deleted "wind" and "thermal" from that mapping, so probably "thermal_value" must be deleted as well from the rest of the parser?! You'll figure it out 😆 

```
station_map = {
     "coal": ["Ukai(3-5)+Ukai6",
             "Wanakbori",
             "Gandhinagar",
             "Sikka(3-4)",
             "KLTPS(1-3)+KLTPS4",
             "SLPP(I+II)",
             "Akrimota",
             "TPAECo  ",
             "EPGL(I+II)",
             "Adani(I+II+III)",
             "BECL(I+II)",
             "CGPL "],
    "hydro": ["Ukai  (Hydro)",
              "Kadana (Hydro)",
              "SSP(RBPH)"],
    "gas": ["Utran(Gas)(II)",
            "Dhuvaran (Gas)(I)+(II)",
            "GIPCL(I)",
            "GIPCL(II)",
            "GSEG(I+II)",
            "GPPC",
            "CLPI",
            "KAWAS",
            "Sugen+ Unosgn",
            "JHANOR"],
    "nuclear": ["KAPP"]   
}
```
When running with the new station map I get the following;
```
(EM-env) chris@ThinkPad:~/electricitymap$ python -m parsers.IN_GJ
Unknown fuel for station name: Ukai  (Hydro)
Unknown fuel for station name: TPAECo
solar returned -1.00, setting to None
{'zoneKey': 'IN-GJ', 'production': {'biomass': None, 'solar': None, 'nuclear': None, 'hydro': 52.0, 'unknown': 438.0, 'coal': 3746.0, 'oil': None, 'gas': 1575.0, 'geothermal': None, 'wind': 1625.0}, 'source': 'sldcguj.com', 'storage': {'hydro': None}, 'datetime': datetime.datetime(2018, 4, 11, 21, 39, 50, tzinfo=tzfile('/usr/share/zoneinfo/Asia/Kolkata'))}
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/chris/electricitymap/parsers/IN_GJ.py", line 172, in <module>
    print(fetch_consumption('IN-GJ', session))
  File "/home/chris/electricitymap/parsers/IN_GJ.py", line 157, in fetch_consumption
    value_map = fetch_data(zone_key, session)
  File "/home/chris/electricitymap/parsers/IN_GJ.py", line 99, in fetch_data
    logger.warning('Unknown fuel for station name: {}'.format(v1),
AttributeError: 'NoneType' object has no attribute 'warning'
```

However that error is nothing to do with the station mapping, on L157 fetch_data is called without a logger argument which results in logger=None. Fixing that I get the following;
```
(EM-env) chris@ThinkPad:~/electricitymap$ python -m parsers.IN_GJ
Unknown fuel for station name: Ukai  (Hydro)
Unknown fuel for station name: TPAECo
solar returned -1.00, setting to None
{'source': 'sldcguj.com', 'production': {'biomass': None, 'wind': 1582.0, 'hydro': 52.0, 'unknown': 436.0, 'gas': 1567.0, 'geothermal': None, 'coal': 3738.0, 'oil': None, 'nuclear': None, 'solar': None}, 'zoneKey': 'IN-GJ', 'storage': {'hydro': None}, 'datetime': datetime.datetime(2018, 4, 11, 21, 48, 50, tzinfo=tzfile('/usr/share/zoneinfo/Asia/Kolkata'))}
Unknown fuel for station name: Ukai  (Hydro)
Unknown fuel for station name: TPAECo
{'consumption': 14112.0, 'zoneKey': 'IN-GJ', 'datetime': datetime.datetime(2018, 4, 11, 21, 48, 50, tzinfo=tzfile('/usr/share/zoneinfo/Asia/Kolkata')), 'source': 'sldcguj.com'}
```

Some of the plants aren't matching due to whitespace issues as you said before, we'll need string manipulation or regex to fix that.
@systemcatch thank you for checking! 😃👍
Looks like around 2 GW of generation go missing in the total sum.
Your values are 7.3 GW in total. When I take a recent snapshot (~1 hour later), I get 9.5 GW total generation.

Resolving patch:
diff --git a/parsers/IN_GJ.py b/parsers/IN_GJ.py
--- a/parsers/IN_GJ.py
+++ b/parsers/IN_GJ.py
@@ -1,29 +1,63 @@
 #!/usr/bin/env python3
 
 import re
-from ast import literal_eval
-from arrow import get
-from requests import Session
-from .lib import zonekey
-from .lib import web
-from .lib import IN
+import collections
+from operator import itemgetter
+import arrow
+import requests
+import itertools
+from .lib import zonekey, web, IN
 from .lib.validation import validate
 from logging import getLogger
-from operator import itemgetter
+
 
 station_map = {
-    "hydro": ["Ukai  (Hydro)", "Kadana (Hydro)", "SSP(RBPH)"],
-    "thermal": ["Ukai(3-5)+Ukai6", "Wanakbori", "Gandhinagar", "Sikka(3-4)", "KLTPS(1-3)+KLTPS4",
-                "SLPP(I+II)", "Akrimota", "JHANOR"],
-    "wind": ["GMR"],
-    "gas": ["Utran(Gas)(II)", "Dhuvaran (Gas)(I)+(II)", "GIPCL(I)", "GIPCL(II)", "GSEG(I+II)", "GPPC", "CLPI",
-            "KAWAS", "Sugen+ Unosgn"],
-    "nuclear": ["KAPP"],
-    "coal": ["TPAECo", "EPGL(I+II)", "Adani(I+II+III)", "BECL(I+II)", "CGPL"]
+     "coal": ["Ukai(1-5)+Ukai6",
+             "Wanakbori",
+             "Gandhinagar",
+             "Sikka(1-2)+Sikka(3-4)",
+             "KLTPS(1-3)+KLTPS4",
+             "SLPP(I+II)",
+             "Akrimota",
+             "TPAECo",
+             "EPGL(I+II)",
+             "Adani(I+II+III)",
+             "BECL(I+II)",
+             "CGPL"],
+    "hydro": ["Ukai(Hydro)",
+              "Kadana(Hydro)",
+              "SSP(RBPH)"],
+    "gas": ["Utran(Gas)(II)",
+            "Dhuvaran(Gas)(I)+(II)",
+            "GIPCL(I)",
+            "GIPCL(II)",
+            "GSEG(I+II)",
+            "GPPC",
+            "CLPI",
+            "KAWAS",
+            "Sugen+Unosgn",
+            "JHANOR"],
+    "nuclear": ["KAPP"]
 }
 
 
+def split_and_sum(expression):
+    """
+    Avoid using literal_eval for simple addition expressions.
+    Returns sum of all positive numbers.
+    """
+
+    split_vals = expression.split('+')
+    float_vals = [float(v) for v in split_vals]
+    total = sum([v for v in float_vals if v > 0.0])
+
+    return total
+
+
 def fetch_data(zone_key, session=None, logger=None):
+    session = session or requests.session()
+
+    values = collections.Counter()
     zonekey.assert_zone_key(zone_key, 'IN-GJ')
 
     solar_html = web.get_response_soup(
@@ -31,63 +65,114 @@ def fetch_data(zone_key, session=None, logger=None):
     wind_html = web.get_response_soup(
         zone_key, 'https://www.sldcguj.com/RealTimeData/wind.php', session)
 
-    india_date = get(solar_html.find_all('tr')[0].text.split('\t')[-1].strip() + ' Asia/Kolkata', 'D-MM-YYYY H:mm:ss ZZZ')
-
-    solar_value = float(literal_eval(solar_html.find_all('tr')[-1].find_all('td')[-1].text.strip()))
-    wind_value = float(literal_eval(wind_html.find_all('tr')[-1].find_all('td')[-1].text.strip()))
-
-    hydro_value = thermal_value = gas_value = coal_value = unknown = 0.0
-
-    value_map = {
-        "date": india_date.datetime,
-        "solar": solar_value,
-        "hydro": hydro_value,
-        "thermal": thermal_value,
-        "wind": wind_value,
-        "gas": gas_value,
-        "coal": coal_value,
-        "unknown": unknown
-    }
+    values['date'] =  arrow.get(
+        solar_html.find_all('tr')[0].text.split('\t')[-1].strip()
+        + ' Asia/Kolkata', 'D-MM-YYYY H:mm:ss ZZZ').datetime
+    values['solar'] = split_and_sum(
+        solar_html.find_all('tr')[-1].find_all('td')[-1].text.strip())
+    values['wind'] = split_and_sum(
+        wind_html.find_all('tr')[-1].find_all('td')[-1].text.strip())
 
     cookies_params = {
         'ASPSESSIONIDSUQQQTRD': 'ODMNNHADJFGCMLFFGFEMOGBL',
         'PHPSESSID': 'a301jk6p1p8d50dduflceeg6l1'
     }
 
-    # other_html = requests.get('https://www.sldcguj.com/RealTimeData/RealTimeDemand.php', params=cookies_params)
-    rows = web.get_response_soup(zone_key, 'https://www.sldcguj.com/RealTimeData/RealTimeDemand.php',
-                                 session).find_all('tr')
+    rows = web.get_response_soup(
+        zone_key,
+        'https://www.sldcguj.com/RealTimeData/RealTimeDemand.php',
+        session).find_all('tr')
 
     for row in rows:
-        if len(row.find_all('td')) > 3:  # will find production rows
-            v1, v2 = (re.sub(r'[\n\t\r]', r'', x.text.strip()) for x in itemgetter(*[0, 3])(row.find_all('td')))
+        elements = row.find_all('td')
+        if len(elements) > 3:  # will find production rows
+            v1, v2 = (re.sub(r'\s+', r'', x.text)
+                      for x in itemgetter(*[0, 3])(elements))
             energy_type = [k for k, v in station_map.items() if v1 in v]
             if len(energy_type) > 0:
-                value_map[energy_type[0]] += float(literal_eval(v2))
+                v2 = split_and_sum(v2)
+                values[energy_type[0]] += v2
             else:
-                if 'Station Name' in (v1, v2):  # meta data row
+                if 'StationName' in (v1, v2):  # meta data row
+                    continue
+                elif 'DSMRate' in v2:  # demand side management
                     continue
                 else:
-                    value_map['unknown'] += float(literal_eval(v2))
-                    logger.warning('Unknown fuel for station name: {}'.format(v1),
+                    try:
+                        logger.warning(
+                            'Unknown fuel for station name: {}'.format(v1),
+                            extra={'key': zone_key})
+                        v2 = split_and_sum(v2)
+                        values['unknown'] += v2
+                    except ValueError as e:
+                        # handle float failures
+                        logger.warning(
+                            "couldn't convert {} to float".format(v2),
+                            extra={'key': zone_key})
+                        continue
+        elif len(elements) == 3:  # will find consumption row
+            v1, v2 = (re.sub(r'\s+', r'', x.text)
+                      for x in itemgetter(*[0, 2])(elements))
+            if v1 == 'GujaratCatered':
+                values['total consumption'] = split_and_sum(v2.split('MW')[0])
+        elif len(elements) == 1:
+            # CGPL/KAPP/KAWAS/JHANOR plants have a different html structure.
+            plant_name = re.sub(r'\s+', r'', elements[0].text)
+            known_plants = itertools.chain.from_iterable(station_map.values())
+
+            if plant_name in known_plants:
+                energy_type = [k for k, v in station_map.items() if plant_name in v][0]
+                generation_tag = row.find_all_next("td")[3]
+                val = float(re.sub(r'\s+', r'', generation_tag.text))
+                if val > 0:
+                    values[energy_type] += val
+            else:
+                if plant_name and plant_name != 'GMR':
+                    # GMR is outside Gujarat, sometimes plant_name is ''
+                    logger.warning(
+                        'Unknown fuel for station name: {}'.format(plant_name),
                         extra={'key': zone_key})
-        elif len(row.find_all('td')) == 3:  # will find consumption row
-            v1, v2 = (re.sub(r'[\n\t\r]', r'', x.text.strip()) for x in itemgetter(*[0, 2])(row.find_all('td')))
-            if v1 == 'Gujarat Catered':
-                value_map['total consumption'] = float(literal_eval(v2.split(' ')[0]))
 
-    return value_map
+    return values
 
 
-def fetch_production(zone_key='IN-GJ', session=None, target_datetime=None, logger=getLogger('IN-GJ')):
+def fetch_production(zone_key='IN-GJ', session=None, target_datetime=None,
+                     logger=getLogger('IN-GJ')):
     """
-    Method to get production data of Gujarat
-    :param zone_key:
-    :param session:
-    :return:
+    Requests the last known production mix (in MW) of a given country
+    Arguments:
+    zone_key: specifies which zone to get
+    session: request session passed in order to re-use an existing session
+    target_datetime: the datetime for which we want production data. If not provided, we should
+      default it to now. The provided target_datetime is timezone-aware in UTC.
+    logger: an instance of a `logging.Logger`; all raised exceptions are also logged automatically
+    Return:
+    A list of dictionaries in the form:
+    {
+      'zoneKey': 'FR',
+      'datetime': '2017-01-01T00:00:00Z',
+      'production': {
+          'biomass': 0.0,
+          'coal': 0.0,
+          'gas': 0.0,
+          'hydro': 0.0,
+          'nuclear': null,
+          'oil': 0.0,
+          'solar': 0.0,
+          'wind': 0.0,
+          'geothermal': 0.0,
+          'unknown': 0.0
+      },
+      'storage': {
+          'hydro': -10.0,
+      },
+      'source': 'mysource.com'
+    }
     """
+    session = session or requests.session()
     if target_datetime:
-        raise NotImplementedError('This parser is not yet able to parse past dates')
+        raise NotImplementedError(
+            'This parser is not yet able to parse past dates')
 
     value_map = fetch_data(zone_key, session, logger=logger)
 
@@ -96,15 +181,15 @@ def fetch_production(zone_key='IN-GJ', session=None, target_datetime=None, logge
         'datetime': value_map['date'],
         'production': {
             'biomass': None,
-            'coal': value_map['coal'],
-            'gas': value_map['gas'],
-            'hydro': value_map['hydro'],
-            'nuclear': None,
+            'coal': value_map.get('coal', 0),
+            'gas': value_map.get('gas', 0),
+            'hydro': value_map.get('hydro', 0),
+            'nuclear': value_map.get('nuclear', 0),
             'oil': None,
-            'solar': value_map['solar'],
-            'wind': value_map['wind'],
+            'solar': value_map.get('solar', 0),
+            'wind': value_map.get('wind', 0),
             'geothermal': None,
-            'unknown': value_map['unknown']
+            'unknown': value_map.get('unknown', 0)
         },
         'storage': {
             'hydro': None
@@ -117,17 +202,20 @@ def fetch_production(zone_key='IN-GJ', session=None, target_datetime=None, logge
     return valid_data
 
 
-def fetch_consumption(zone_key='IN-GJ', session=None, target_datetime=None, logger=getLogger('IN-GJ')):
+def fetch_consumption(zone_key='IN-GJ', session=None, target_datetime=None,
+                      logger=getLogger('IN-GJ')):
     """
     Method to get consumption data of Gujarat
     :param zone_key:
     :param session:
     :return:
     """
+    session = session or requests.session()
     if target_datetime:
-        raise NotImplementedError('This parser is not yet able to parse past dates')
+        raise NotImplementedError(
+            'This parser is not yet able to parse past dates')
 
-    value_map = fetch_data(zone_key, session)
+    value_map = fetch_data(zone_key, session, logger=logger)
 
     data = {
         'zoneKey': zone_key,
@@ -140,6 +228,6 @@ def fetch_consumption(zone_key='IN-GJ', session=None, target_datetime=None, logg
 
 
 if __name__ == '__main__':
-    session = Session()
+    session = requests.Session()
     print(fetch_production('IN-GJ', session))
     print(fetch_consumption('IN-GJ', session))


Test patch: