Repository: electricitymaps/electricitymaps-contrib

Issue:
Add Maharashtra (India) IN-MH
Not sure if any of you OCR-experts already had a go at it in the past, but here is another challenge still waiting for you ;)
I tried to elaborate a bit more on what needs to be included and where to find this info in the images.
Feel free to comment below and reach out on Slack should you have any questions related to this. 😸 
_ _ _ _

This one may have gone lost a bit in the large #304 "Add India" (and #817 OCR) thread and could cover yet another large state on the west coast of India. So I decided to open a seperate issue on this and see what we can do here for **Maharashtra**.
_ _ _ _

On the [website of Maharashtra State Load Despatch Center](https://mahasldc.in/) there is a tab named "SCADA Data".

First off, there seems to be a **single image containing all of the relevant generation** for all types:
I played around with the URLs of the various images, and changing the end of the URL to "3.jpg" leads to this major dashboard:
https://mahasldc.in/wp-content/reports/sldc/mvrreport3.jpg

* **coal:** "THERMAL" in the lower left part of the image is for generation from coal. There is data for net generation ("behind bus") and gross generation, I suggest using the latter one to cover self-consumption related emissions (~+10%).
* **gas:** Below thermal, there is "GAS". Don't get distracted by the "Uran" label, it's not related to nuclear but just a city name of the unit's location south of Mumbai.
* **hydro:** The next column slightly left of the center has a field summarizing total "hydro" output.
  - Note: Within the hydro units, there is ["Ghatghar", which is a **pumped storage hydro** plant](https://en.wikipedia.org/wiki/Ghatghar_Dam)!!! If we decide to put this in its respective category, there is another field "Ghatgr pump" for the pumping consumption (I have not yet observed if it's showing any value, it's 0 at the moment.)
So the Ghatghar output could be subtracted from HYDRO and added to hydro storage, in case OCR works reliably enough to perform this.
* **solar, wind, biomass, unknown:** On the bottom of this column, there is a yellow box containing several types of generation.
Obviously, MS WIND is **wind** and MS SOLAR is **solar**. CO-GEN is from **biomass** ([bagasse](https://en.wikipedia.org/wiki/Bagasse)). 
I'd map the output of of "OTHER + SMHYD" entirely to "**unknown**", because small hydro has a tiny share (scroll down to the end for the direct link to that dashboard).
* **nuclear:** There are nuclear units of Tarapur power station TARPR PH-I (oldest operational in the world) and TARPR PH-II, displayed under the "C. S. GENERATION" section on the very right

There are three more crucial things to consider, which makes it really complicated.
1. The city of Mumbai, which is within Maharashtra on the west coast, has a seperate box with figures for generation and demand. It is split to "TPC THM" for thermal, "TPC HYD" for hydro and "REL GEN", probably for "relative share of generation in some power plants outside of Mumbai". 
1.1 TPC THM could be added to coal, because these are the types of plants in Mumbai operated by Tata Power Company (=TPC!). 
1.2 There is no hydro power plant directly within the city of Mumbai. But this figure is probably related to some hydro power plants of the private sector, which also belong to the private Mumbai based Tata Power Company with 310 MW of hydro installed very, very close to Mumbai in three locations. https://npp.gov.in/dashBoard/cp-map-dashboard So I think it's legit to include these hydro figures as well! 
1.3 "REL GEN" could be excluded or just added to unknown.
2. There is another huge chunk of "IPP / CPP" generation, which stands for Intependent and Central power producers. The list contains 9 of such power plants plus "other", that seem to produce for Maharashtra, but are not necessarily located within the state. I tried to locate them and their fuel types:

* JINDAL - [Jindal Tamnar **coal** power plant](https://en.wikipedia.org/wiki/Jindal_Tamnar_Thermal_Power_Plant) in **Chhattisgarh**
* ADANI - [Mundra thermal **coal** power station operated by Adani](https://en.wikipedia.org/wiki/Mundra_Thermal_Power_Station) in **Gujarat** (we've had this in the last Gujarat improvement issue)
* IDEAL ENR - [Bela **coal** power plant owned by ideal](https://www.industryabout.com/country-territories-3/757-india/fossil-fuels-energy/42622-bela-tata-coal-power-plant) in **Maharashtra**
* RATTAN_IND-AMST and _IND-NSK - [RattanIndia **coal** power plants](https://en.wikipedia.org/wiki/RattanIndia) in Amravati and Nasik in **Maharashtra**
* BUTIBORI-REL - [coal power plant](https://en.wikipedia.org/wiki/Butibori_Power_Project) in **Maharashtra**
* WARDHA - [Wardha-Warora coal power plant](https://en.wikipedia.org/wiki/Wardha_Warora_Power_Plant) in **Maharashtra**
* DHARIWAL - [Dhariwal **coal** power station](https://en.wikipedia.org/wiki/Dhariwal_Power_Station) in **Maharashtra**
* PIONEER - [Pioneer **gas** power plant](http://www.pgpl.in/) in **Maharashtra**

3. Central state generation is also displayed on the right side of the board. Some of these ultra super large power plants are in Maharashtra, some are outside of it. Not sure what to do with this data, maybe including all that is within IN-MH is the way to go since they feed in the grid of this state and make up the mix.
Here are the power plants of the central sector and their types + locations:
* KORBAN - [Korba **coal** power station](https://en.wikipedia.org/wiki/Korba_Super_Thermal_Power_Station) in **Chhattisgarh**
* VINCHAL - [Vindhyachal **coal** power station](https://en.wikipedia.org/wiki/Vindhyachal_Thermal_Power_Station) in **Madhya Pradesh**
* GANDHAR - [Jhanor-Ghandar **gas** power station](https://en.wikipedia.org/wiki/Jhanor-Gandhar_Thermal_Power_Station) in **Gujarat**
* KAWAS - [Kawas **coal** power station](https://en.wikipedia.org/wiki/Kawas_Thermal_Power_Station) in **Gujarat**
* KK' PARA - [Kakrapar **nuclear** power station](https://en.wikipedia.org/wiki/Kakrapar_Atomic_Power_Station) in **Gujarat**
* TARPR PH-I and TARPR PH-II - [Tarapur **nuclear**](https://en.wikipedia.org/wiki/Tarapur_Atomic_Power_Station)  in **Maharashtra (see comments above)**
* SIPAT - [Sipat **coal** power station](https://en.wikipedia.org/wiki/Sipat_Thermal_Power_Station) in **Chhattisgarh**
* SSD - [Sardar Sarovar **hydro** dam](https://www.water-technology.net/projects/sardar-sarovar-dam-gujarat/) in **Gujarat** (suffering from drouhgts)
* RGPPL - [Ratnagiri **gas** and power ](https://en.wikipedia.org/wiki/Ratnagiri_Gas_and_Power) in **Maharashtra**
* MAUDA STG-I and STG-II - [Mauda super thermal **coal** power station](https://en.wikipedia.org/wiki/Mauda_Super_Thermal_Power_Station) in **Maharashtra**
* CGPL MUNDRA - [Mundra ultra mega **coal** power plant](https://en.wikipedia.org/wiki/Mundra_Ultra_Mega_Power_Plant) in **Gujarat**
* SOLPRN - [Solapur super thermal **coal** power station](https://en.wikipedia.org/wiki/Solapur_Super_Thermal_Power_Station) in **Maharashtra**

Here is the image of the dashboard with all that info:

![image](https://user-images.githubusercontent.com/25743609/40913902-8361adf2-67f6-11e8-9c02-abfc83627d46.png)





_ _ _ _
Further details for the types of generation, but not necessarily needed for the parser:

**Station wise generation, exchange and demand** (many, many figures):
https://mahasldc.in/report2/
https://mahasldc.in/wp-content/reports/sldc/report2.jpg

![image](https://user-images.githubusercontent.com/25743609/40913606-6ba4b250-67f5-11e8-81f3-a49cfb8d5be9.png)

**Wind generation data:**
https://mahasldc.in/on-line-re-data/
https://mahasldc.in/wp-content/reports/sldc/mvrreport5.jpg

![image](https://user-images.githubusercontent.com/25743609/40913977-bc8deb72-67f6-11e8-9f8b-572601796176.png)

**Solar data:**
https://mahasldc.in/on-line-solar-data/
https://mahasldc.in/wp-content/reports/sldc/mvrreport8.jpg

![image](https://user-images.githubusercontent.com/25743609/40914075-124336a8-67f7-11e8-8a2f-5411679e0c58.png)

Something called **"[Bagasse data" (= biomass)](https://en.wikipedia.org/wiki/Bagasse)**:
https://mahasldc.in/on-line-co-gen-others-data/
https://mahasldc.in/wp-content/reports/sldc/mvrreport7.jpg

![image](https://user-images.githubusercontent.com/25743609/40914128-48501324-67f7-11e8-88df-66011b602d49.png)

And, finally, **"others and small hydro"**:
https://mahasldc.in/on-line-others-small-hydro/
https://mahasldc.in/wp-content/reports/sldc/mvrreport9.jpg

_ _ _ _ 

There are some more dashboards with **power line schemes and line loadings**, but this is wayyyy to complex:

https://mahasldc.in/wp-content/reports/sldc/mvrreport2.jpg
https://mahasldc.in/wp-content/reports/sldc/mvrreport4.jpg
https://mahasldc.in/wp-content/reports/sldc/mvrreport6.jpg

![image](https://user-images.githubusercontent.com/25743609/41111999-ec81aae2-6a7d-11e8-8997-0a09026b4378.png)

![image](https://user-images.githubusercontent.com/25743609/41111990-e7322d28-6a7d-11e8-83ba-2eb0d9dc44ae.png)

![image](https://user-images.githubusercontent.com/25743609/41112018-02e38e7c-6a7e-11e8-8c66-7151b8869483.png)



Discussion:
That's a good challenge!

I ran tesseract on the main image but unsurprisingly got garbage back. We could split the dashboard into multiple images using the pillow module to try and improve OCR performance. Or we could do the same on the smaller thermal/solar/wind images which will be easier to work with.
Using [Daily Reports](https://mahasldc.in/home.php/daily-reports/) one can fetch (previous)day data in PDF. That might help.
Hi, 
I'm new to electricity map but I'm a big fan and I figured to put the knowledge from my recent image processing lecture to good use and build the parser!

You can check out the code here: https://github.com/GitMatze/maharashtra_parser
For now it's just a demo but it seems to work well and reliably. 
Here's some of the data I've been logging over the last hours: 

![logs](https://user-images.githubusercontent.com/45139485/75094554-767e6a00-558c-11ea-8c64-02d8a7fe8151.png)

If you can make use of that, I can integrate this into electricityMap. However I would need some further information on how to exactly use all these figures. I figured out how to build the total demand and production sums but the Central State figures in the righmost column remain kind of unclear to me. Since the figure MAH./DRAWL in the upper right table corresponds to CS EXCH in the middle section I suppose that that's the share of the total Central State production that goes into Maharashtra State Demand. To account for that I think we have to calculate the mix of the central state production in order to then break down CS EXCH..? That would mean reading in a lot of numbers though. 

Here are the notes I took to understand the picture, maybe they are helpful: 
![Anmerkung 2020-02-22 155454](https://user-images.githubusercontent.com/45139485/75094695-08d33d80-558e-11ea-8f58-88c4e0e127de.png)





@GitMatze this is just incredible, good job! 👍 👍 👍 

The wind/solar/hydro data in your first shows perfect expected behaviour of the renewables.
The conventional fuels look good as well. I'll try to have another look at the dashboard this weekend to make sure what is needed/not needed for the electricityMap.
Thanks!

I drew a quick overview of what's going on there, at least to my understanding.
![grafik](https://user-images.githubusercontent.com/45139485/75095681-c282dc00-5597-11ea-8b37-e0540b537aa5.png)


Fantastic work @GitMatze!

From looking over your code I don't see a problem adapting it to work with the map.

Resolving patch:
diff --git a/parsers/IN_MH.py b/parsers/IN_MH.py
new file mode 100644
--- /dev/null
+++ b/parsers/IN_MH.py
@@ -0,0 +1,270 @@
+from PIL import Image, ImageOps
+import pytesseract
+import cv2
+from imageio import imread
+import numpy as np
+import arrow
+import logging
+
+url = 'https://mahasldc.in/wp-content/reports/sldc/mvrreport3.jpg'
+
+# specifies locations of data in the image
+# (x,y,x,y) = upper left, lower right corner of rectangle
+locations = {
+    'MS WIND' : {
+        'label' : (595,934,692,961),
+        'value' : (785,934,844,959)
+    },
+    'MS SOLAR' : {
+            'label' : (595,963,705,984),
+            'value' : (785,959,848,983)
+    },
+    'THERMAL' : {
+            'label' : (407,982,502,1004),
+            'value' : (516,987,584,1015)
+    },
+    'GAS' : {
+            'label' : (403,1033,493,1056),
+            'value' : (515,1038,582,1068)
+    },
+    'HYDRO' : {
+            'label' : (589,472,666,496),
+            'value' : (753,472,817,494)
+    },
+    'TPC HYD.' : {
+            'label' : (926,525,1035,554),
+            'value' : (1100,527,1173,551)
+    },
+    'TPC THM.' : {
+            'label' : (924,578,1030,604),
+            'value' : (1088,581,1173,605)
+    },
+    'COGEN' : {
+            'label' : (594,989,670,1011),
+            'value' : (789,982,844,1007)
+    },
+    'OTHR+SMHYD' : {
+            'label' : (594,1009,730,1031),
+            'value' : (789,1004,846,1032)
+    },
+    'AEML GEN.' : {
+            'label' : (922,687,1041,716),
+            'value' : (1081,692,1175,717)
+    },
+    'CS GEN. TTL.' : {
+            'label' : (1341,998,1492,1029),
+            'value' : (1549,1001,1616,1023)
+    },
+    'KK’ PARA' : {
+            'label' : (1346,708,1457,730),
+            'value' : (1560,711,1626,732)
+    },
+    'TARPR PH-I' : {
+            'label' : (1341,730,1484,757),
+            'value' : (1556,733,1626,755)
+    },
+    'TARPR PH-II' : {
+            'label' : (1341,750,1490,782),
+            'value' : (1556,757,1626,781)
+    },
+    'SSP' : {
+            'label' : (1341,800,1401,823),
+            'value' : (1557,802,1626,825)
+    },
+    'RGPPL' : {
+            'label' : (1343,822,1421,849),
+            'value' : (1560,820,1620,850)
+    },
+    'GANDHAR' : {
+            'label' : (1341,660,1446,689),
+            'value' : (1562,656,1623,685)
+    },
+    'CS EXCH' : {
+            'label' : (920,303,1021,338),
+            'value' : (1090,309,1172,334)
+    },
+    # STATE DEMAND (including Mumbai!)
+    'DEMAND' : {
+            'label' : (932,1003,1021,1030),
+            'value' : (1080,996,1167,1018)
+    },
+    # RE TTL
+    'TTL' : {
+            'label' : (597,1035,663,1056),
+            'value' : (783,1032,845,1056)
+    },
+    'TTL (IPP/CPP+RE)' : {
+            'label' : (594,1072,765,1098),
+            'value' : (786,1068,847,1093)
+    },
+    'SOLAR TTL' : {
+            'label' : (592,577,715,605),
+            'value' : (772,583,814,602)
+    },
+    'PIONEER' : {
+            'label' : (592,910,694,929),
+            'value' : (794,910,844,927)
+    }
+}
+
+generation_map = {
+    'biomass' : {
+        'add': ['COGEN'],
+        'subtract': []
+    },
+    'coal' : {
+        'add': ['THERMAL',
+                'TTL (IPP/CPP+RE)',
+                'TPC THM.',
+                'CS GEN. TTL.',
+                'AEML GEN.',
+                'SOLAR TTL'],
+        'subtract': ['TTL',
+                     'PIONEER',
+                     'SSP',
+                     'RGPPL',
+                     'TARPR PH-I',
+                     'TARPR PH-II',
+                     'KK’ PARA',
+                     'GANDHAR']
+    },
+    'gas': {
+        'add': ['GAS',
+                'PIONEER',
+                'GANDHAR',
+                'RGPPL'],
+        'subtract': []
+    },
+    'hydro' : {
+        'add': ['HYDRO',
+                'TPC HYD.',
+                'SSP'],
+        'subtract': []
+    },
+    'nuclear' : {
+        'add': ['TARPR PH-I',
+                'TARPR PH-II',
+                'KK’ PARA'],
+        'subtract': []
+    },
+    'solar' : {
+        'add': ['MS SOLAR'],
+        'subtract': []
+    },
+    'wind' : {
+        'add': ['MS WIND'],
+        'subtract': []
+    },
+    'unknown' : {
+        'add': ['OTHR+SMHYD'],
+        'subtract': []
+    }
+}
+
+# list of values that belong to Central State production
+CS = ['SSP',
+      'RGPPL',
+      'TARPR PH-I',
+      'TARPR PH-II',
+      'KK’ PARA',
+      'GANDHAR',
+      'CS GEN. TTL.']
+
+# converts image into a black and white
+def RGBtoBW(pil_image):
+    # pylint: disable=no-member
+    image = cv2.cvtColor(np.array(pil_image), cv2.COLOR_RGB2GRAY)
+    image = cv2.threshold(image, 0, 255,
+                          cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
+    return Image.fromarray(image)
+
+# performs text recognition on given location and source image
+def recognize(location, source, lang):
+    img = source.crop(location)
+    img = RGBtoBW(img)
+    img = ImageOps.invert(img)
+    text = pytesseract.image_to_string(img, lang=lang, config='--psm 7')
+
+    return text, img
+
+# ignores highly indistinguishable characters
+def areEqual(str1,str2):
+    chars = {
+        '{':'(',
+        '}': ')',
+        'T': 'L',
+        'I': 'L',
+        'P': 'E',
+        ' ': ''
+    }
+    for key, value in chars.items():
+        str1 = str1.replace(key,value)
+        str2 = str2.replace(key,value)
+    return str1.lower() == str2.lower()
+
+
+# TODO: this function actually fetches consumption data
+def fetch_production(zone_key='IN-MH', session=None, target_datetime = None,
+                     logger=logging.getLogger(__name__)):
+
+    if target_datetime is not None:
+        raise NotImplementedError('This parser is not yet able to parse past dates')
+
+    dt = arrow.now('Asia/Kolkata').floor('minute').datetime
+
+    data = {
+        'zoneKey': 'IN-MH',
+        'datetime': dt,
+        'production': {
+            'biomass': 0.0,
+            'coal': 0.0,
+            'gas': 0.0,
+            'hydro': 0.0,
+            'nuclear': 0.0,
+            'solar': 0.0,
+            'wind': 0.0,
+            'unknown': 0.0
+        },
+        'storage': {},
+        'source': 'mahasldc.in',
+    }
+
+    image = imread(url)
+    image = Image.fromarray(image)  # create PIL image
+
+    labels = {}
+    values = {}
+
+    # read in label-value pairs from the image as specified in locations dict
+    for type, locs in locations.items():
+        label,_ = recognize(locs['label'], image, 'eng')
+        value, _ = recognize(locs['value'], image, 'digits_comma')
+        labels[type] = label
+        values[type] = max( [float(value),0] )
+
+        assert (areEqual(label,type)), \
+            'Wrongly regognized label `{}` as `{}`'.format(type, label)
+
+    # fraction of central state production that is exchanged with Maharashtra
+    share = values['CS EXCH'] / values['CS GEN. TTL.']
+
+    for type, plants in generation_map.items():
+        for plant in plants['add']:
+            fac = share if plant in CS else 1  # add only a fraction of central state plant consumption
+            data['production'][type] += fac * values[plant]
+        for plant in plants['subtract']:
+            fac = share if plant in CS else 1
+            data['production'][type] -= fac * values[plant]
+
+    # Sum over all production types is expected to equal the total demand
+    demand_diff = sum( data['production'].values() ) - values['DEMAND']
+    assert (abs( demand_diff) < 30), \
+        'Production types do not add up to total demand. Difference: {}'.format(demand_diff)
+
+    return data
+
+
+if __name__ == '__main__':
+
+    print( fetch_production() )
+    
\ No newline at end of file


Test patch: