Repository: electricitymaps/electricitymaps-contrib

Issue:
Broken CY capacity parser
## Description
Cyprus capacity is broken, caused by a change in the website. This change happened before the one that needed my previous PR #5702.
I can fix this, but I need some guidance as to what does the backend expect. Should it return the capacity when asked for any date, or only for the current time? Also, why is the capacity and production data returned together, instead of separated? In this case being returned in different calls would be more reasonable, in my opinion. Does the backend support this?


Discussion:
> I can fix this, but I need some guidance as to what does the backend expect. Should it return the capacity when asked for any date, or only for the current time?

Only return the capacity if the capacity for that particular time is available, otherwise omit it.

> Also, why is the capacity and production data returned together, instead of separated?

Just legacy infrastructure, we are working on it but it has not been a priority.

> In this case being returned in different calls would be more reasonable, in my opinion. Does the backend support this?

Agreed, but the backend does not support this, _yet._
> Only return the capacity if the capacity for that particular time is available, otherwise omit it.

Ok, but capacity is only available in the real time page. How do I know if i am asked for the current time data? I've seen that when no time is given all parsers assume current time, however when I manually run the parser it is always given a date, even if I don't use --target_datetime. Does the backend also always give a date?
> > Only return the capacity if the capacity for that particular time is available, otherwise omit it.
> 
> Ok, but capacity is only available in the real time page. How do I know if i am asked for the current time data? 

If the backend wants realtime data it will call the parser without a target datetime. Then it's up to the parser to determine how that is handled but usually it means the most recent data should be returned.

> I've seen that when no time is given all parsers assume current time, however when I manually run the parser it is always given a date, even if I don't use --target_datetime. Does the backend also always give a date?

It should not be given a date but it should return objects that have valid dates. But depending how the parser is done it might create it's own target_datetime if the target_datetime that is passed to it are None.



Resolving patch:
diff --git a/parsers/CY.py b/parsers/CY.py
--- a/parsers/CY.py
+++ b/parsers/CY.py
@@ -12,6 +12,9 @@
 from bs4 import BeautifulSoup
 from requests import Session
 
+REALTIME_SOURCE = "https://tsoc.org.cy/electrical-system/total-daily-system-generation-on-the-transmission-system/"
+HISTORICAL_SOURCE = "https://tsoc.org.cy/electrical-system/archive-total-daily-system-generation-on-the-transmission-system/?startdt={}&enddt=%2B1days"
+
 
 class CyprusParser:
     CAPACITY_KEYS = {
@@ -33,7 +36,7 @@ def warn(self, text: str) -> None:
 
     def parse_capacity(self, html) -> dict:
         capacity = {}
-        table = html.find(id="production_graph_static_data")
+        table = html.find(id="production_graph_static_data2")
         for tr in table.find_all("tr"):
             values = [td.string for td in tr.find_all("td")]
             key = self.CAPACITY_KEYS.get(values[0])
@@ -83,13 +86,13 @@ def parse_production(self, html, capacity: dict) -> list:
 
     def fetch_production(self, target_datetime: Optional[datetime]) -> list:
         if target_datetime is None:
-            url = "https://tsoc.org.cy/electrical-system/total-daily-system-generation-on-the-transmission-system/"
+            url = REALTIME_SOURCE
         else:
             # convert target datetime to local datetime
             url_date = (
                 arrow.get(target_datetime).to("Asia/Nicosia").format("DD-MM-YYYY")
             )
-            url = f"https://tsoc.org.cy/electrical-system/archive-total-daily-system-generation-on-the-transmission-system/?startdt={url_date}&enddt=%2B1days"
+            url = HISTORICAL_SOURCE.format(url_date)
 
         res = self.session.get(url)
         assert (
@@ -98,7 +101,8 @@ def fetch_production(self, target_datetime: Optional[datetime]) -> list:
 
         html = BeautifulSoup(res.text, "lxml")
 
-        capacity = self.parse_capacity(html)
+        # Capacity is only available if we fetch data from realtime url
+        capacity = self.parse_capacity(html) if url is REALTIME_SOURCE else {}
         data = self.parse_production(html, capacity)
 
         if len(data) == 0:
@@ -115,11 +119,8 @@ def fetch_production(
     """Requests the last known production mix (in MW) of a given country."""
     assert zone_key == "CY"
 
-    target_datetime = target_datetime or datetime.utcnow()
-
     parser = CyprusParser(session or Session(), logger)
-    if isinstance(target_datetime, datetime):
-        return parser.fetch_production(target_datetime)
+    return parser.fetch_production(target_datetime)
 
 
 if __name__ == "__main__":


Test patch: