Repository: electricitymaps/electricitymaps-contrib

Issue:
Data for Israel
Another one from me this weekend :)

Israel has data for the "system load", which is updated every few minutes. This is also equivalent to "system production", because there are no imports (only exports are shown on IEA's statistics)

This url https://www.iec.co.il/_layouts/iec/applicationpages/lackmanagment.aspx leads to this data table:

![image](https://user-images.githubusercontent.com/25743609/69496293-e9fed580-0ed0-11ea-8168-bdc43bf0aee5.png)

I thought from the html of the page, it would be easy to get the desired "statusVal":
`<span id="lblActualSupply"><div class="IecContentTitle">צריכת החשמל כעת במגוו"ט</div><span class="statusVal">8,067</span></span>`
 
But here is a part of what I get back as response instead:

```
<Response [200]>
<html>
 <head>
  <meta charset="utf-8"/>
  <script>
   window.rbzid="F1MPV4aQjOIsxeqssncm3guiYZhNzK1pbkx9JHjc36L9ok7CIWB5gwQE01ukWP8TcKv8MwNyWRltsisX49d+Ty1Znd7yQzLgKv2NkuVd6ew=";
        u82222.O=function(x){return x;};u82222.E=function (){return typeof u82222.t.u==='function'?
...
...
...
E(21)]){}else{location[u82222.f(u82222.r(11)+u82222.x(6)+u82222.e(17)+u82222.N(0),u82222.e(2))]();}
  </script>
 </head>
 <body>
 </body>
</html>
```

So looks like it actually returns some sort of script.

I need some help from you folks to create the parser ;) 
Could somebody please tell me how to get the desired part from the html of the website using BeautifulSoup or sth similar? Thanks! :) 


Discussion:
It looks like the first time we make the request, you get the weird obfuscated thing, but also a cookie. When you redo the request with the cookie, you get plain HTML.

```
>>> import requests
>>> first = requests.get('https://www.iec.co.il/_layouts/iec/applicationpages/lackmanagment.aspx')
>>> first.text[:200]
'<html><head><meta charset="utf-8"><script>\n        window.rbzid="J36FVmfMyCuT4whUvFe57EfnnF/zvwbiIetiFbCG8ciKfDDVxa7AjytlugMJTM7CP9DP0OffLsOjblPYcwaJ11t4AmdUyCK48muUVE00o2k=";\n        u82222.O=functio'
>>> first.cookies
<RequestsCookieJar[Cookie(version=0, name='rbzid', value='J36FVmfMyCuT4whUvFe57EfnnF/zvwbiIetiFbCG8ciKfDDVxa7AjytlugMJTM7CP9DP0OffLsOjblPYcwaJ11t4AmdUyCK48muUVE00o2k=', port=None, port_specified=False, domain='.iec.co.il', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False), Cookie(version=0, name='GCLB', value='CPuawNfln7q5Gg', port=None, port_specified=False, domain='www.iec.co.il', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)]>
>>> second = requests.get('https://www.iec.co.il/_layouts/iec/applicationpages/lackmanagment.aspx', cookies=first.cookies)
>>> second.text
'\r\n\r\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> \r\n<html xmlns="http://www.w3.org/1999/xhtml">\r\n<head><title>\r\n\tחברת החשמל לישראל\r\n</title><link id="cssLink" rel="stylesheet" type="text/css" href="/Style Library/Iec/he-IL/LackManagment.css" />\r\n    <script type="text/javascript">\r\n    </script>\r\n</head>\r\n<body style="margin:0;" aria-owns="miflas-heading IecStatus">\r\n    <dl id="IecStatus" class="iecStatus" role="list">\r\n\t\t\t<dt id="miflas-heading" role="heading" aria-level="4">\r\n                מפלס החשמל\r\n            </dt>\r\n\t\t\t<dd class="current" role="listitem">\r\n\t\t\t\t<h6 role="presentation">    \r\n                    <span id="lblActualSupply"><div class=\'IecContentTitle\'>צריכת החשמל כעת במגוו"ט</div><span class=\'statusVal\'>8,290</span></span>     \r\n                </h6>\r\n\t\t    </dd>\r\n\t\t\t<dd class="pic" role="listitem">\r\n\t\t\t\t<h6 role="presentation">\r\n                    <span id="lblHours"><div class=\'IecContentTitle\'>שעת השיא</div><span class=\'statusVal\'>18:00</span></span>\r\n\t\t\t\t</h6>\r\n\t\t\t</dd>\r\n\t\t\t<dd class="resCurrent" role="listitem">  \r\n                <img id="imgBattery" class="resCurrentPic" src="/Style%20Library/Iec/he-il/Images/LackManagment/BatteryLevel_1000.png" border="0" />\r\n\t\t\t\t<h6 role="presentation">  \r\n                    <span id="lblCurrentSupplyTitle">רזרבת החשמל כעת במגוו"ט</span>\r\n                    <span id="lblCurrentSupply"><span class=\'statusVal\'>2,314</span></span>\r\n                </h6>\r\n\t\t\t</dd>\r\n\t\t\t<dd class="picRes" role="listitem">\r\n\t\t\t\t<h6 role="presentation"> \r\n                    <span id="lblSupply"><div class=\'IecContentTitle\'>רזרבת החשמל בשיא במגוו"ט</div><span class=\'statusVal\'>1,611</span></span>\r\n                </h6>\r\n\t\t\t</dd>\r\n\t\t</dl>\r\n \r\n\r\n        <span id="miflas">2</span> \r\n        \r\n            <script type="text/javascript" >\r\n                setTimeout("location.reload(true);", 120000);\r\n        </script>\r\n        \r\n    </body>\r\n</html>\r\n'
```
Looks like the breakdown of capacity (https://en.wikipedia.org/wiki/Energy_in_Israel#Electricity) is heavily fossil fuel based with little renewable energy. Since there won't be any exports/imports we can estimate the CO2 intensity.

Resolving patch:
diff --git a/parsers/IL.py b/parsers/IL.py
new file mode 100644
--- /dev/null
+++ b/parsers/IL.py
@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+# coding=utf-8
+
+# This parser returns Israel's electricity system load (assumed to be equal to electricity production)
+# Source: Israel Electric Corporation
+# URL: https://www.iec.co.il/en/pages/default.aspx
+# Shares of Electricity production in 2018: 65.6% oil, 34.4% gas (source: IEA; https://www.iea.org/data-and-statistics?country=ISRAEL&fuel=Electricity%20and%20heat&indicator=Electricity%20generation%20by%20source)
+
+
+import arrow
+import requests
+import re
+from bs4 import BeautifulSoup
+
+
+def fetch_production(zone_key='IL', session=None, logger=None):
+    first = requests.get('https://www.iec.co.il/_layouts/iec/applicationpages/lackmanagment.aspx')
+    first.cookies
+    second = requests.get('https://www.iec.co.il/_layouts/iec/applicationpages/lackmanagment.aspx', cookies=first.cookies)
+
+    soup = BeautifulSoup(second.content, 'lxml')
+
+    span = soup.find("span", attrs={"class": "statusVal"})
+    value = re.findall("\d+", span.text.replace(",", ""))
+    load = int(value[0])
+    production = {}
+    production['unknown'] = load
+    
+    datapoint = {
+        'zoneKey': zone_key,
+        'datetime': arrow.now('Asia/Jerusalem').datetime,
+        'production': production,
+        'source': 'iec.co.il'
+    }
+
+    return datapoint
+
+
+if __name__ == '__main__':
+    """Main method, never used by the Electricity Map backend, but handy for testing."""
+    
+    print('fetch_production() ->')
+    print(fetch_production())


Test patch: