Repository: electricitymaps/electricitymaps-contrib

Issue:
Data for Armenia AM
There is a generation **schedule** for Armenia with data for hydro, gas and nuclear:

It is in the PDF in the second link on this side (ends with aktiv.pdf)

http://www.energyoperator.am/information.html

I ran the PDF through an OCR converter for fun do decipher the first column. Worked perfectly for letters and numbers, just the "-" signs had to be replaced.

I attached an [Excel file](https://github.com/tmrowco/electricitymap/files/1833243/armenia.xlsx) with the outcomes and some edits to highlight the rows of interest.

Here is what the result looks like:

![image](https://user-images.githubusercontent.com/25743609/37710411-dc194f1e-2d0d-11e8-8e66-739028c5bdc6.png)


Even if this is only a day-ahead schedule, it could be worth downloading the PDF each day just to have them, since there seems to be no data for the previous days or any historical data.


Discussion:
Do you think it's worth implementing in order to get the "last 24h" graphs working? 
As long as there is nothing that is "more live", the day-ahead schedule is the only option for AM now.
But because this is in conflict with being as close to live as possible, I'd say *NO*.

If there was an option to highlight countries with scheduled data (striped/dottet/etc.) in the future, let's come back to this.
I will close this as no real-time or recently-retrospective data appears to be available
@jarek Can we reopen this one?
Real-time data seems to be available from Electro Power Systems Operator at:
http://epso.am/scada.htm
![Armenia_Scada](https://user-images.githubusercontent.com/25774514/55820851-c71bde80-5b04-11e9-92f5-1f2e16bc3500.png)
What is included:
Top left, in blue: 
- Սպառում - Consumption
- Գեներացիա - Generation
- Փոխհոսք - Exchanges, Positive number=Export?

Boxes top row:
- left: ԱԷԿ - nuclear units and total in yellow
- second from left, upper: Hrazdan 5, gas
- second from left, lower: Yerevan Thermal Power Plant, gas
- middle upper: Hrazdan TPP 1-4, gas
- middle bottom row: Total thermal
- right bottom right corner row: Total hydro

Bottom row, left column: Միջհամ փոխհոսքեր - Interdependencies
- Again, positive number=export?
- Ահար -Ahar:Iran, two cables listed together and then separately
- Ալավեիդի - Alaverdi:Georgia?
- Արցախ - Artsakh: Officially Azerbaijan(?)
- Շինուհայր - Shinuhayr: no idea
- Լալվար - Lalvar: Georgia?
- Մեղրհ - Meghr(i):Iran?



Yep, and the numbers are different now than in your screenshot, so we know the data is not leftover static.

It's all one big image and the contrast between text and background colour isn't great, so I don't know how successful OCR will be, but might be worth a shot for someone.

Edit: to counter the problems of trying to OCR a complex table data layout, it might be worth it to first crop the image to sub-parts that only contain one data column, then run the OCR.
Great work @tmslaine!

There looks to be exchange info at http://epso.am/poweren.htm

![AM_ex](https://user-images.githubusercontent.com/30196510/55869087-b5c2e880-5b7d-11e9-9d65-897f90813553.png)

Could we manually check if the data on http://epso.am/poweren.htm matches the data on http://epso.am/scada.htm a couple of times a day? The former is text so would be a lot easier to integrate. I guess it doesn't break out the thermal power plants nor "alternative station" by fuel though.
FYI: I am working on this parser atm and will submit a work-in-progress PR soon ;)

Resolving patch:
diff --git a/parsers/AM.py b/parsers/AM.py
new file mode 100644
--- /dev/null
+++ b/parsers/AM.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+# coding=utf-8
+
+import arrow
+import re
+import requests
+from datetime import datetime
+from dateutil import parser as dparser, tz
+from bs4 import BeautifulSoup
+
+#URL of the power system summary: http://epso.am/poweren.htm
+#URL of the detailled SCADA-page: http://epso.am/scada.htm
+
+power_plant_mapping = {
+    'var atom': 'nuclear', # atom = 'atomnaya elektro stantsiya'
+    'var tes': 'gas',      # tes = 'termalnaya elektro stantsiya' - only gas in AM
+    'var ges': 'hydro',    # ges = 'gidro elektro stantsiya'
+    'var altern': 'hydro'  # altern = two hydro power plants according to SCADA data (middle row, right box)
+    }
+
+tie_line_mapping = {
+    'var Lalvar': 'GE',
+    'var ninoc': 'GE',
+    'var alaver': 'GE',
+    'var shin': 'AZ(AM)',
+    'var arcakh': 'AZ(AM)',
+    'var ahar': 'IR'
+    }
+
+other_variables_mapping = {
+    'var cons': 'total production', #please note, this is an error of the variable name ('cons' is actually used for total production)
+    'var peretok': 'total import/export', #positive = import, negative = export
+    'var herc2': 'frequency',       #"hertz" ;)
+    'var time2': 'timestamp',       #date is missing
+    'sparum2': 'total consumption', #please note, this is an error of the variable name (probably means "production", but is used for consumption)
+    }
+
+soup_content_variables_mapping = {
+    '[0]' : 'empty',
+    '[1]' : 'Lalvar (GE)',
+    '[2]' : 'ninoc (GE)',
+    '[3]' : 'alaver (GE)',
+    '[4]' : 'shin (AZ)',
+    '[5]' : 'arcakh (AZ)',
+    '[6]' : 'ahar (IR)',
+    '[7]' : 'cons [total production]',
+    '[8]' : ' altern',
+    '[9]' : 'atom',
+    '[10]' : 'tes',
+    '[11]' : 'ges',
+    '[12]' : 'peretok',
+    '[13]' : 'herc2',
+    '[14]' : 'time2',
+    '[15]' : 'sparum2 []',
+    '[16]' : 'empty',
+    '[17]' : 'empty',
+    }
+
+def fetch_production(zone_key='AM', session=None, target_datetime=None, logger=None):
+    r = session or requests.session()
+    url = 'http://epso.am/poweren.htm'
+    response = r.get(url)
+    response.encoding = 'utf-8'
+    html_doc = response.text
+    soup = BeautifulSoup(html_doc, 'html.parser')
+
+    data_string = soup.find(text=re.compile('var'))
+    data_split = data_string.split('\r\n')
+
+    gas_tes = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[10])
+    gas_total = float(gas_tes[0])
+
+    hydro_ges = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[11])
+    hydro_altern = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[8])
+    hydro_total = float(hydro_ges[0])+float(hydro_altern[0])
+
+    nuclear_atom = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[9])
+    nuclear_total = float(nuclear_atom[0])
+
+    yerevan = tz.gettz('Asia/Yerevan')
+    date_time = dparser.parse(data_split[14].split()[3], default=datetime.now(yerevan), fuzzy=True)
+
+    #Operating solar, wind and biomass plants exist in small numbers, but are not reported yet
+    data = {
+        'zoneKey': zone_key,
+        'datetime': date_time,
+        'production': {
+            'gas': gas_total,
+            'hydro': hydro_total,
+            'nuclear': nuclear_total,
+            'biomass': None,
+            'coal': 0,
+            'geothermal': 0,
+            'oil': 0,
+            'solar': None,
+            'wind': None
+            },
+        'storage': {
+            'hydro': 0,
+            'battery': 0
+            },
+        'source': 'http://epso.am/poweren.htm'
+    }
+
+    return data
+
+
+def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, logger=None):
+    """Requests the last known power exchange (in MW) between two countries
+    Arguments:
+    zone_key (optional) -- used in case a parser is able to fetch multiple countries
+    session (optional)      -- request session passed in order to re-use an existing session
+    Return:
+    A dictionary in the form:
+    {
+      'sortedZoneKeys': 'DK->NO',
+      'datetime': '2017-01-01T00:00:00Z',
+      'netFlow': 0.0,
+      'source': 'mysource.com'
+    }
+    """
+    sorted_keys = '->'.join(sorted([zone_key1, zone_key2]))
+
+    r = session or requests.session()
+    url = 'http://epso.am/poweren.htm'
+    response = r.get(url)
+    response.encoding = 'utf-8'
+    html_doc = response.text
+    soup = BeautifulSoup(html_doc, 'html.parser')
+
+    data_string = soup.find(text=re.compile('var'))
+    data_split = data_string.split('\r\n')
+    GE_1 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[1])
+    GE_2 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[2])
+    GE_3 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[3])
+    AZ_1 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[4])
+    AZ_2 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[5])
+    IR_1 = re.findall("[-+]?[.]?[\d]+(?:,\d\d\d)*[\.]?\d*(?:[eE][-+]?\d+)?", data_split[6])
+
+    AM_AZ = float(AZ_1[0])+float(AZ_2[0])
+    AM_GE = float(GE_1[0])+float(GE_2[0])+float(GE_3[0])
+    AM_IR = float(IR_1[0])
+
+    yerevan = tz.gettz('Asia/Yerevan')
+    date_time = dparser.parse(data_split[14].split()[3], default=datetime.now(yerevan), fuzzy=True)
+
+
+    if sorted_keys == 'AM->AZ':
+        netflow = -1 * AM_AZ
+    elif sorted_keys == 'AM->GE':
+        netflow = -1 * AM_GE
+    elif sorted_keys == 'AM->IR':
+        netflow = -1 * AM_IR
+    else:
+        raise NotImplementedError('This exchange pair is not implemented')
+
+    exchange = {
+        'sortedZoneKeys': sorted_keys,
+        'datetime': date_time,
+        'netFlow': netflow,
+        'source': 'http://epso.am/poweren.htm'
+    }
+    return exchange
+
+if __name__ == '__main__':
+    print('fetch_production->')
+    print(fetch_production())
+    print('fetch_exchange(AM, AZ) ->')
+    print(fetch_exchange('AM', 'AZ'))
+    print('fetch_exchange(AM, GE) ->')
+    print(fetch_exchange('AM', 'GE'))
+    print('fetch_exchange(AM, IR) ->')
+    print(fetch_exchange('AM', 'IR'))


Test patch: