Repository: electricitymaps/electricitymaps-contrib

Issue:
Bulgaria is missing



Discussion:
Belgium and Italy have also been missing for a while.
As far as I can see in the [BG parser](https://github.com/tmrowco/electricitymap/blob/master/parsers/BG.py) the values are obtained from tso.bg, which is no longer an active domain. Instead the Bulgarian Operator is currently housed under eso.bg, with the operational data provided here: http://www.eso.bg/?did=124 . Looking at the python code, the parser looks for a table that looks identical to the one in the active link, from which it extracts data to send to the website.
In my opinion simply changing the url would fix the problem, but I am not competent in Python to verify that, if somebody could help that would be greatly appreciated.
Ah I thought Bulgaria was part of the ENTSOE parser but clearly it's separate.

@stgram I quickly tried the new url in the parser but it throws errors with the datetime.  Looking at the page source the data is still in a table but certain parts have changed enough that altering the parser is needed.  If you wanted to have a go at fixing it we could help out if you get stuck, shouldn't need any extra modules or more complex code.
@systemcatch Thank you for the response. Unfortunately I am not familiar with Py, I only understand the general concept of the code. Looking at the [source of the page](http://www.eso.bg/?did=124) the datetime is on line 81:
```
<div class='dashboardDiv'>
<div class='dashboardCaptionDiv'>Генерация и товар на ЕЕС за <b>
04.12.2017 20:53:14  </b></div>
<center><small style='color:#555;'>
```

The parser used to look for a tag with the id lblTecDate, which does not exist anymore.
```
datetime = None
    local_time = soup.find(id='lblTecDate')
    if local_time:
        time = local_time.string.split(':')
        if len(time) == 3:
            datetime = arrow.now('Europe/Sofia').floor('day').\
                replace(hour=int(time[0]), minute=int(time[1]), second=int(time[2]))
    if not datetime:
        raise Exception('No datetime')
```
Instead would it be possible to fetch the content of the div tag, which includes the whole date & time information? I found this [documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#id12) on the issue, searching for the bulgarian text "Генерация и товар на ЕЕС за" in the tag should return the whole tag, from which the datetime could be extracted, or perhaps a regex expression for the datetime format? I cannot understand how the arrow commands work.
Please help me with the issue, if you need assistance with the bulgarian language I am here to translate.

I would do the following to find the tag.  The div class attribute is not unique so we also search by text.  Then we find the b tag and get the text within. 

```python
div = soup.find('div', 'class' = 'dashboardCaptionDiv', text = 'Генерация и товар на ЕЕС за')
b = div.find('b')
time_string = b.contents
```
We can use the [strptime method](https://docs.python.org/2.7/library/datetime.html#datetime.datetime.strptime) to convert the time string into a datetime object.
Could somebody try this? Sounds like a simple thing, lets get one more country on the map!

Resolving patch:
diff --git a/parsers/BG.py b/parsers/BG.py
--- a/parsers/BG.py
+++ b/parsers/BG.py
@@ -1,11 +1,9 @@
 # encoding=utf8
-# The arrow library is used to handle datetimes
+
 import arrow
-# The request library is used to fetch content through HTTP
-import requests
 from bs4 import BeautifulSoup
+import requests
 
-TIME_ZONE = 'Europe/Sofia'
 TYPE_MAPPING = {                        # Real values around midnight
     u'АЕЦ': 'nuclear',                  # 2000
     u'Кондензационни ТЕЦ': 'coal',      # 1800
@@ -19,6 +17,13 @@
     u'Товар РБ': 'consumption',         # 3175
 }
 
+def time_string_converter(ts):
+    """Converts time strings into aware datetime objects."""
+
+    dt_naive = arrow.get(ts, 'DD.MM.YYYY HH:mm:ss')
+    dt_aware = dt_naive.replace(tzinfo = 'Europe/Sofia').datetime
+
+    return dt_aware
 
 def fetch_production(country_code='BG', session=None):
     """Requests the last known production mix (in MW) of a given country
@@ -50,65 +55,44 @@ def fetch_production(country_code='BG', session=None):
       'source': 'mysource.com'
     }
     """
+
     r = session or requests.session()
-    url = 'http://www.tso.bg/GeneratedPowersPublicExt/'
+    url = 'http://www.eso.bg/?did=124'
     response = r.get(url)
     html = response.text
     soup = BeautifulSoup(html, 'html.parser')
 
-    # first compute the datetime
-    datetime = None
-    local_time = soup.find(id='lblTecDate')
-    if local_time:
-        time = local_time.string.split(':')
-        if len(time) == 3:
-            datetime = arrow.now('Europe/Sofia').floor('day').\
-                replace(hour=int(time[0]), minute=int(time[1]), second=int(time[2]))
-
-    if not datetime:
-        raise Exception('No datetime')
-
-    # then populate an object that contains the consumptions by type, we will parse a table something like that
-    # АЕЦ	1998
-    # Кондензационни ТЕЦ	1790
-    # Топлофикационни ТЕЦ	146
-    # Заводски ТЕЦ	146
-    # ВЕЦ	7
-    # Малки ВЕЦ	72
-    # ВяЕЦ	483
-    # ФЕЦ	0
-    # Био ТЕЦ	30
-    # Товар РБ	3114
-
-    res = dict()
-    lines = soup.find_all('table')         # each line is in a <table>
-    for l in lines:
-        try:    # we want to read only values that have a non empty label and a non empty float value
-            label = l.find('td')
-            value = l.find('span')
-            if value and label:
-                energy_type = TYPE_MAPPING.get(label.string.strip(), 'unknown')
-                res[energy_type] = res.get(energy_type, 0.0) + float(value.string)
-        except (TypeError, ValueError), e:
-            print str(e)
-            pass    # this ensure we skip lines that have no float value
+    try:
+        time_div = soup.find("div", {"class": "dashboardCaptionDiv"})
+        bold = time_div.find('b')
+    except AttributeError:
+        raise LookupError('No data currently available for Bulgaria.')
+
+    time_string = bold.string
+    dt = time_string_converter(time_string)
+
+    table = soup.find("table", {"class": "defaultTable2"})
+    rows = table.findChildren("tr")
+
+    datapoints = []
+    for row in rows[1:-1]:
+        gen_tag = row.find("td")
+        gen = gen_tag.text
+        val_tag = gen_tag.findNext("td")
+        val = float(val_tag.find("b").text)
+        datapoints.append((TYPE_MAPPING[gen], val))
+
+    production = {}
+    for k,v in datapoints:
+        production[k] = production.get(k, 0.0) + v
 
     data = {
-        'countryCode': 'BG',
-        'production': {
-            'biomass': res.get('biomass', 0.0),
-            'hydro': res.get('hydro', 0.0),
-            'nuclear': res.get('nuclear', 0.0),
-            'wind': res.get('wind', 0.0),
-            'coal': res.get('coal', 0.0),
-            'solar': res.get('solar', 0.0),
-            'gas': res.get('gas', 0.0),
-            'unknown': res.get('unknown', 0.0)
-        },
+        'countryCode': country_code,
+        'production': production,
         'storage': {},
-        'source': 'tso.bg',
-        'datetime': datetime.datetime
-    }
+        'source': 'eso.bg',
+        'datetime': dt
+        }
 
     return data
 


Test patch: