Repository: electricitymaps/electricitymaps-contrib

Issue:
Namibia parser should read out data timestamp
Current time in Namibia is 2 April 14:41 but the [Nampower dashboard](http://www.nampower.com.na/Scada.aspx) is stuck on 1 April 11:45:40.

Timestamp of the data is available in http://www.nampower.com.na/gis/images/File_Info.png

We could OCR that file. A quick attempt using the current tesseract methods and upscaling from 600x20 to 1200x40 seems to have worked alright, I got `SCADA File Dene: 01/04/2018 11:45:40` (it should have been "File Date" but otherwise correct). There is always the danger of OCR misreading, so maybe only ignore the datapoint if the timestamp can be read and is too far in the past.

Screenshot from our page:
![na](https://user-images.githubusercontent.com/47415/38196680-b4fa6000-3684-11e8-95a4-9c03b6a19576.png)




Discussion:
That's annoying, similar issue with the SG solar ocr stalling but still returning data that was accepted. I think I tried to do something with the timestamp but never got it to read properly.

edit: do you think it's better for the parser to fail if the timestamp can't be read? That way we avoid having the scada dashboard get stuck on an image whose timestamp can't be parsed correctly, which will result in the data being accepted anyway.

Resolving patch:
diff --git a/parsers/NA.py b/parsers/NA.py
--- a/parsers/NA.py
+++ b/parsers/NA.py
@@ -12,6 +12,7 @@
 import requests
 
 
+timestamp_link = 'http://www.nampower.com.na/gis/images/File_Info.png'
 generation_link = 'http://www.nampower.com.na/gis/images/Gx.png'
 exchanges_link = 'http://www.nampower.com.na/gis/images/Imports_Exports.png'
 
@@ -39,11 +40,8 @@ def get_text_from_image(link, expected_size, new_size, logger, session=None):
     img = Image.open(s.get(link, stream=True).raw)
 
     if img.size != expected_size:
-        if (logger):
-            logger.warning("Check Namibia Scada dashboard for {} changes.".format(link),
-                extras={'key': 'NA'})
-        else:
-            print("Check Namibia Scada dashboard for {} changes.".format(link))
+        logger.warning("Check Namibia Scada dashboard for {} changes.".format(link),
+            extras={'key': 'NA'})
 
     gray = img.convert('L')
     gray_enlarged = gray.resize(new_size, Image.LANCZOS)
@@ -52,6 +50,35 @@ def get_text_from_image(link, expected_size, new_size, logger, session=None):
     return text
 
 
+def check_timestamp(session=None, logger=None):
+    """
+    Sometimes the Scada Dashboard image stops updating for a while.
+    This function tries to ensure that only data younger than 1 hour
+    is accepted.
+    """
+
+    scada_info = get_text_from_image(session=session, link=timestamp_link,
+                                     expected_size=(600,20), new_size=(1200,40),
+                                     logger=logger)
+
+    timestamp = scada_info.split(':', 1)[1]
+
+    try:
+        scada_time = arrow.get(timestamp, ' DD/MM/YYYY HH:mm:ss')
+    except arrow.parser.ParserError as e:
+        logger.warning('Namibia scada timestamp cannot be read, got {}.'.format(timestamp))
+        # The OCR of the Scada dashboard is not very reliable, on failure safer to assume data is good.
+        return
+
+    data_time = scada_time.replace(tzinfo='Africa/Windhoek')
+    current_time = arrow.now('Africa/Windhoek')
+    diff = current_time - data_time
+
+    # Need to be sure we don't get old data if image stops updating.
+    if diff.seconds > 3600:
+        raise ValueError('Namibia scada data is too old to use, data is {} hours old.'.format(diff.seconds/3600))
+
+
 def data_processor(text):
     """
     Takes text produced from OCR and extracts production.
@@ -107,6 +134,7 @@ def fetch_production(zone_key = 'NA', session=None, target_datetime=None, logger
                                    logger=logger)
 
     production = data_processor(raw_text)
+    check_timestamp(session=session, logger=logger)
 
     data = {
           'zoneKey': zone_key,
@@ -177,6 +205,8 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
     if flow is not None:
         flow = -1 * flow
 
+    check_timestamp(session=session, logger=logger)
+
     exchange = {'sortedZoneKeys': sorted_codes,
                 'datetime': arrow.now('Africa/Windhoek').datetime,
                 'netFlow': flow,


Test patch: