Repository: electricitymaps/electricitymaps-contrib

Issue:
Data available for French Polynesia
Hi, first issue here, hope I'm not doing it the wrong way!

Real-time production/consumption data for _most_ of the islands of French Polynesia is available here : [https://www.edt.pf/transition-energetique-innovation](url)
Production = consumption since there isn't any exchange with neighboring countries/dependent territories.
Electricity in French Polynesia is only generated by thermal (oil), hydro and solar plants. There used to be wind turbines on a few islands, and there are projects to conduct some other experimentation, but it's not clear if they currently are in operation.

The data above-mentioned covers the electricity generated by EDT/Engie and its subsidiary Marama Nui on the most populated islands (including Tahiti, Moorea, Huahine, Bora Bora) where at least 75-80% of the population lives :
- [https://www.edt.pf/edt-engie](url)
- [https://www.edt.pf/documents/20126/1180028/EDT-chiffres+2018-10.jpg/10e3828d-6eb3-5c82-ab6b-ede4055cfb74?t=1582934078733imagePreview=1](url).

The production on the remaining islands is generated by domestic stations operated by municipalities. I've mapped most of the stations, even on the remote islands using many different sources, don't know if this can prove useful to estimate the total installed capacity for French Polynesia.

There's also historic data available here with a 10-minute update (XLSX files) : [https://www.service-public.pf/sde/documentation-2/production-delectricite-edt-sur-tahiti/](url). Unfortunately it only covers the production on the island of Tahiti (~2/3 of the population).

I've started to write a parser for the data using guidance from the Wiki, but I doubt the method I used is optimal. I have basic knowledge in coding (but am very interested in learning more), so I was wondering if anyone could mentor me on this one.

Cheers :)





Discussion:
Hey @autipial, I'd be happy to work together on implementing this parser if you like :)
Hey @vakili, thanks for the reply ! I'll reach out to you on Slack.

Resolving patch:
diff --git a/parsers/PF.py b/parsers/PF.py
new file mode 100644
--- /dev/null
+++ b/parsers/PF.py
@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+
+import logging
+import requests
+import lxml
+from bs4 import BeautifulSoup
+import re
+import json
+import arrow
+
+TZ = 'Pacific/Tahiti'
+
+def fetch_production(
+    zone_key='PF',
+    session=None,
+    target_datetime=None,
+    logger: logging.Logger = logging.getLogger(__name__),
+):
+    """Requests the last known production mix (in MW) of a given country
+    Arguments:
+    ----------
+    zone_key: used in case a parser is able to fetch multiple countries
+    session: request session passed in order to re-use an existing session
+    target_datetime: the datetime for which we want production data. If not
+      provided, we should default it to now. If past data is not available,
+      raise a NotImplementedError. Beware that the provided target_datetime is
+      UTC. To convert to local timezone, you can use
+      `target_datetime = arrow.get(target_datetime).to('America/New_York')`.
+      Note that `arrow.get(None)` returns UTC now.
+    logger: an instance of a `logging.Logger` that will be passed by the
+      backend. Information logged will be publicly available so that correct
+      execution of the logger can be checked. All Exceptions will automatically
+      be logged, so when something's wrong, simply raise an Exception (with an
+      explicit text). Use `logger.warning` or `logger.info` for information
+      that can useful to check if the parser is working correctly. A default
+      logger is used so that logger output can be seen when coding / debugging.
+    Returns:
+    --------
+    If no data can be fetched, any falsy value (None, [], False) will be
+      ignored by the backend. If there is no data because the source may have
+      changed or is not available, raise an Exception.
+    A dictionary in the form:
+    {
+      'zoneKey': 'FR',
+      'datetime': '2017-01-01T00:00:00Z',
+      'production': {
+          'biomass': 0.0,
+          'coal': 0.0,
+          'gas': 0.0,
+          'hydro': 0.0,
+          'nuclear': null,
+          'oil': 0.0,
+          'solar': 0.0,
+          'wind': 0.0,
+          'geothermal': 0.0,
+          'unknown': 0.0
+      },
+      'storage': {
+          'hydro': -10.0,
+      },
+      'source': 'mysource.com'
+    }
+    """
+    r = session or requests.Session()
+    if target_datetime is None:
+        url = "https://www.edt.pf/transition-energetique-innovation"  
+    else:
+        # WHEN HISTORICAL DATA IS NOT AVAILABLE
+        raise NotImplementedError('This parser is not yet able to parse past dates')
+    
+    res = r.get(url)
+    assert res.status_code == 200, 'Exception when fetching production for ' \
+                                   '{}: error when calling url={}'.format(zone_key, url)
+    
+    soup = BeautifulSoup(res.text, 'lxml')
+    block = soup.find(id="id1____detailMesurePortlet__WAR__EDTAEL2018__Script").prettify()
+    block = block.replace('\n','')
+    block = re.search(r'\{"cols.*\}\]\}\]\}', block)
+
+    data_table = block.group(0)
+    data_table = re.sub('Thermique', 'oil', data_table)
+    data_table = re.sub('Hydro électricité', 'hydro', data_table)
+    data_table = re.sub('Solaire', 'solar', data_table)
+            
+    data_dict = json.loads(data_table)
+
+    data = {
+        'zoneKey': zone_key,
+        'datetime': arrow.utcnow().floor('minute').datetime,
+        'production': {
+        'biomass': 0.0,
+        'coal': 0.0,
+        'gas': 0.0,
+        'hydro': None,
+        'nuclear': 0.0,
+        'oil': None,
+        'solar': None,
+        'wind': None,
+        'geothermal': 0.0,
+        'unknown': None
+        },
+        'storage': {},
+        'source': 'edt.pf'
+    }
+
+    # Parse the dict 'data_dict' containing the production mix (the values are in kW)
+    for line in data_dict['rows']:
+        resource = line['c'][0].get('v')
+        production_value = line['c'][1].get('v')
+        production_value_mw = production_value/1000
+        data['production'][resource] = production_value_mw
+        
+    return data
+
+if __name__ == '__main__':
+    """Main method, never used by the Electricity Map backend, but handy
+    for testing."""
+
+    print('fetch_production() ->')
+    print(fetch_production())


Test patch: