Repository: electricitymaps/electricitymaps-contrib

Issue:
Chile (CL-SING): total generation stuck at fixed value
The total generation of the CL-SING zone is stuck at 1 939 MW. The renewable part varies, but the "unknown" category always fills the gap. The issue is on the data provider's side.

![image](https://user-images.githubusercontent.com/25743609/39404467-282d9276-4b94-11e8-9472-7cb3a589d39d.png)

![image](https://user-images.githubusercontent.com/25743609/39404501-a6889878-4b94-11e8-9aca-3e25307c52ee.png)
Chile (CL-SING): total generation stuck at fixed value
The total generation of the CL-SING zone is stuck at 1 939 MW. The renewable part varies, but the "unknown" category always fills the gap. The issue is on the data provider's side.

![image](https://user-images.githubusercontent.com/25743609/39404467-282d9276-4b94-11e8-9472-7cb3a589d39d.png)

![image](https://user-images.githubusercontent.com/25743609/39404501-a6889878-4b94-11e8-9aca-3e25307c52ee.png)


Discussion:
@corradio can you see how far back this goes? Hopefully not much more than a few days.
@maxbellec can you help here?
started 2018-04-26 and hasn't happened before. Ignore the the white lines that shouldn't be here

![cl-sing](https://user-images.githubusercontent.com/7253902/39640535-8e559b8a-4fcb-11e8-96cd-b6d5fba31a84.png)
![cl-sing 2](https://user-images.githubusercontent.com/7253902/39640536-8e6e30aa-4fcb-11e8-953f-90fbf13e07ff.png)

I will send them an email about this.

EDIT: email sent, awaiting a response.
On Fri, 4 May 2018, 17:47 maxbellec, <notifications@github.com> wrote:

> started 2018-04-26 and hasn't happened before. Ignore the the white lines
> that shouldn't be here
>
> [image: cl-sing]
> <https://user-images.githubusercontent.com/7253902/39640535-8e559b8a-4fcb-11e8-96cd-b6d5fba31a84.png>
> [image: cl-sing 2]
> <https://user-images.githubusercontent.com/7253902/39640536-8e6e30aa-4fcb-11e8-953f-90fbf13e07ff.png>
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/tmrowco/electricitymap/issues/1356#issuecomment-386660515>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AczDHqwoHRVilBwHLW-jO91kJTd_WZaDks5tvIYTgaJpZM4Trttg>
> .
>

This is still an issue unfortunately
Looks like the problem has been fixed!

![cl_working](https://user-images.githubusercontent.com/30196510/40713780-b99d58fa-63f8-11e8-88a9-466d2c67e185.png)

Happening again in CL-SING:

![image](https://user-images.githubusercontent.com/25743609/41090843-814533da-6a45-11e8-877f-8755d22fabb4.png)

Historical data is available, so this shouldn't be a problem in general, though.

Question: the full breakdown for yesterday's production is already available - is it possible to fill/override that part of the graph with these data?

http://cdec2.cdec-sing.cl/pls/portal/cdec.pck_web_coord_elec.sp_pagina?p_id=5172
Back to normal again, in answer to your question @alixunderplatz I don't see why not.
Feel free to re-open if it happens again!
![image](https://user-images.githubusercontent.com/47415/45596759-7ce4db80-b9c1-11e8-89ac-409c7f7c2022.png)

Honestly I'm not sure what we can do to prevent this. Maybe discard data in the parser if data for more than a few datapoints have the exact same sum?

I also noticed that https://sger.coordinador.cl is an under-construction page. The main domain has some content like https://www.coordinador.cl/sistema-informacion-publica/portal-de-operaciones/operacion-real/generacion-real-de-las-centrales/ but currently that only goes up to 2018-09-05 and also seems country-wide?
@corradio can you see how far back this goes? Hopefully not much more than a few days.
@maxbellec can you help here?
started 2018-04-26 and hasn't happened before. Ignore the the white lines that shouldn't be here

![cl-sing](https://user-images.githubusercontent.com/7253902/39640535-8e559b8a-4fcb-11e8-96cd-b6d5fba31a84.png)
![cl-sing 2](https://user-images.githubusercontent.com/7253902/39640536-8e6e30aa-4fcb-11e8-953f-90fbf13e07ff.png)

I will send them an email about this.

EDIT: email sent, awaiting a response.
On Fri, 4 May 2018, 17:47 maxbellec, <notifications@github.com> wrote:

> started 2018-04-26 and hasn't happened before. Ignore the the white lines
> that shouldn't be here
>
> [image: cl-sing]
> <https://user-images.githubusercontent.com/7253902/39640535-8e559b8a-4fcb-11e8-96cd-b6d5fba31a84.png>
> [image: cl-sing 2]
> <https://user-images.githubusercontent.com/7253902/39640536-8e6e30aa-4fcb-11e8-953f-90fbf13e07ff.png>
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/tmrowco/electricitymap/issues/1356#issuecomment-386660515>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AczDHqwoHRVilBwHLW-jO91kJTd_WZaDks5tvIYTgaJpZM4Trttg>
> .
>

This is still an issue unfortunately
Looks like the problem has been fixed!

![cl_working](https://user-images.githubusercontent.com/30196510/40713780-b99d58fa-63f8-11e8-88a9-466d2c67e185.png)

Happening again in CL-SING:

![image](https://user-images.githubusercontent.com/25743609/41090843-814533da-6a45-11e8-877f-8755d22fabb4.png)

Historical data is available, so this shouldn't be a problem in general, though.

Question: the full breakdown for yesterday's production is already available - is it possible to fill/override that part of the graph with these data?

http://cdec2.cdec-sing.cl/pls/portal/cdec.pck_web_coord_elec.sp_pagina?p_id=5172
Back to normal again, in answer to your question @alixunderplatz I don't see why not.
Feel free to re-open if it happens again!
![image](https://user-images.githubusercontent.com/47415/45596759-7ce4db80-b9c1-11e8-89ac-409c7f7c2022.png)

Honestly I'm not sure what we can do to prevent this. Maybe discard data in the parser if data for more than a few datapoints have the exact same sum?

I also noticed that https://sger.coordinador.cl is an under-construction page. The main domain has some content like https://www.coordinador.cl/sistema-informacion-publica/portal-de-operaciones/operacion-real/generacion-real-de-las-centrales/ but currently that only goes up to 2018-09-05 and also seems country-wide?

Resolving patch:
diff --git a/parsers/CL_SING.py b/parsers/CL_SING.py
--- a/parsers/CL_SING.py
+++ b/parsers/CL_SING.py
@@ -7,7 +7,7 @@
 from __future__ import print_function
 import arrow
 from bs4 import BeautifulSoup
-from collections import defaultdict
+from collections import defaultdict, Counter
 import datetime
 import json
 from pytz import timezone
@@ -203,6 +203,16 @@ def fetch_production(zone_key='CL-SING', session=None, target_datetime=None, log
         }
         production_mix_by_hour.append(production_mix)
 
+    # if production is the same for at least 3 in the last 48 hours, it's
+    # considered constant. Using round because of floating-point precision.
+    c = Counter([round(sum(v for _, v in e['production'].items()), 3)
+                 for e in production_mix_by_hour])
+    most_common = c.most_common(1)[0]
+    if most_common[1] > 3:
+        raise ValueError(
+            'Detected constant prod in CL-SING. Value {} occured {} times in '
+            'the last 48 hours.'.format(most_common[0], most_common[1]))
+
     return production_mix_by_hour
 
 


Test patch: