Repository: electricitymaps/electricitymaps-contrib

Issue:
NL Down
ENTSO-E data seems fine, so is there a problem with energieopwek.nl?


Discussion:
I'm fairly certain it's our parsers. Based on the logs, it seems it's hitting an error
Here's the error by the way: https://kibana.electricitymap.org/app/kibana#/doc/96f67170-0c49-11e9-85c1-1d63df8c862c/logstash-2021.03.01/doc?id=teX77HcBipTrw-Fajpub&_g=(refreshInterval:('$$hashKey':'object:232',display:'5%20minutes',pause:!f,section:2,value:300000),time:(from:now-24h,mode:quick,to:now))
This is because energieopwek.nl does no longer supply us with json data, the response is simply empty:

https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01

As their website describes that we need to contact one of their partners like netanders.nl for API functionality I think they may have closed this down, I don't think their partners offer this for free though

Their regular data.php is encrypted (afaik), but as this gets decoded in the client we could probably reverse engineer this. Not sure if I like that solution though!

I'll send them a message, maybe they just broke the json functionality in an update.
FYI @MartienVisser 

We are having trouble loading JSON data from https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01

Let me know if you can help.
Dear Kenneth,

I know that a new release was done last week. I forward your E-mail to Sander Huizinga from NetAnders. I hope he can help.

Martien

________________________________
From: Kenneth Skovhus <notifications@github.com>
Sent: 01 March 2021 12:01
To: tmrowco/electricitymap-contrib <electricitymap-contrib@noreply.github.com>
Cc: Visser BM, Martien <b.m.visser@pl.hanze.nl>; Mention <mention@noreply.github.com>
Subject: Re: [tmrowco/electricitymap-contrib] NL Down (#3015)


FYI @MartienVisser<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMartienVisser&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092360340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CuqS6dKI2RAooD3sv99AQ55zt3AtksrOwq4pY9ZufDs%3D&reserved=0>

We are having trouble loading JSON data from https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fenergieopwek.nl%2FjsonData.php%3Fsid%3D2ecde3%26Day%3D2021-03-01&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092360340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7sJlhH5ZT12r3mOJaS4dpo%2FKTwOo96%2B%2FTP3%2BMI3JuXI%3D&reserved=0>

Let me know if you can help.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftmrowco%2Felectricitymap-contrib%2Fissues%2F3015%23issuecomment-787859780&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092370330%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Y%2Fqq3s6vQEcYCJAWAqslFfK6lq1Pg9A6G8Q7B%2BVodO4%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMGCWZIVGREPB6SER6PAHHLTBNXZVANCNFSM4YK7FKKQ&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092370330%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HPHyOftlf%2BTUPP2lKKM%2FQENZbbJmPHPDUjHxmumbktg%3D&reserved=0>.

Received message back from them. I was indeed right with my guess that they made scraping more difficult and obfuscated on purpose

They also told me they're also in contact with Tomorrow themselves about it, so I'll let them handle it further.
Thanks for the info. I hope you will resolve it. Martien

________________________________
From: Bart S. <notifications@github.com>
Sent: 01 March 2021 16:36
To: tmrowco/electricitymap-contrib <electricitymap-contrib@noreply.github.com>
Cc: Visser BM, Martien <b.m.visser@pl.hanze.nl>; Mention <mention@noreply.github.com>
Subject: Re: [tmrowco/electricitymap-contrib] NL Down (#3015)


Received message back from them. I was indeed right with my guess that they made scraping more difficult and obfuscated on purpose

They also told me they're also in contact with Tomorrow themselves about it, so I'll let them handle it further.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftmrowco%2Felectricitymap-contrib%2Fissues%2F3015%23issuecomment-788041203&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7Cb38ccbdec44d4b24ad5f08d8dcc7c6bf%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637502097887364966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DtPrSbFR6zdhHUzc62ArG7ZKVwDrDoS1gdPX%2Fw%2Bsh90%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMGCWZK3JEHYFZ5ETC3X7MTTBOX7TANCNFSM4YK7FKKQ&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7Cb38ccbdec44d4b24ad5f08d8dcc7c6bf%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637502097887374964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bIevBePPQcmJMpXB%2FDFs7fRVt3K262eq0JG4XwRZ%2FhI%3D&reserved=0>.

Do we want a parser that works and just uses the ENTSO-E solar and wind (which is much to low) and just puts the rest as unknown?

That can be done fairly easily as a stop-gap until we have a better data source.
> Do we want a parser that works and just uses the ENTSO-E solar and wind (which is much to low) and just puts the rest as unknown?
> 
> That can be done fairly easily as a stop-gap until we have a better data source.

We might have to do this, but let's hold off a bit until we are sure it's the only option as this would affect the historical data quite a bit.
@Kongkille Could we just throw an error or return an empty object for old dates?
It seems that energieopwek can be disabled by passing `energieopwek_nl=False` at https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/NL.py#L15. Is that a config that can be set to disable things for the time being so there's at least some data - albeit incomplete - to show?
We'll know more early next week about what energieopwek wants from us for their data. Based on this, we'll decide what is the best course of action!
https://energieopwek.nl/data.php?sid=2ecde3&Day=2021-03-16 produces correct data and that is what my change #3044 fixes.
Correction, json data seems scrambled. However decoding should be visible in the javascript from the website. Who wants to give that a shot?

The data is still missing? Does anyone know if NetAnders replied and/or shouldn't we set energieopwek_nl to False by default to fix this?
I opt for yes to disable energieopwek for now until a solution presents
Hi everyone, thanks for the activity here! We are working on a solution for this and are hopefully able to share an update soon. For now, we probably don't want to use the fallback, as it is a bit too inaccurate and might cause more confusion. 
Is there any news on when the netherlands is going to be available again?
Hi everyone. 👋  We are working on a permanent solution here, but unfortunately we haven't been able to prioritize this right now. I hope that we will get NL back on the map soon. 🤞 
is there a timeline on when its going to be available again?
Hi everyone, sorry for the long delay, we are now finally prioritising NL 🇳🇱 

Unfortunately, the only __reliable, live, free and hourly available data is from ENTSO-E__, which as we have seen in the past does not completely account for all production (due to some gas/oil/biomass plants not transmitting to the main grid handled by the TSO, Tennet), especially regarding solar and wind (same, production and distribution is handled without integrating with the main TSO grid). 

We nevertheless know that the total reported consumption from ENTSO-E is reliable, and that we can infer from it, and the exchanges, the missing production, which we can then account for adequately in `unknown` (see #2804 ).

As we don't have access to energieopwek's estimates anymore for solar and wind production, we propose the following updates to NL:

1. Accept the underreported solar and wind data from ENTSO-E.
2. Keep computing the missing production, added to `unknown` from the difference with total consumption and exchanges.
3. Take a new emission factor that is a weighted average of the previous `unknown` emission factor with the missing solar and wind production, computed as the difference between ENTSO-E reported 2020 production for both modes and the 2020 production reported by the [central bureau of statistics](https://opendata.cbs.nl/statline/#/CBS/en/dataset/84575ENG/table?dl=45964)
4. Add a disclaimer to the NL zone on the map explaining that live production from solar and wind are underreported but are accounted for as an aggregate in the unknown production. 


Computing the new emission factor:

| Mode    | Emission factor (gCO2eq/kWh) | 2020 ENTSO-E total production (GWh) | 2020 True total production (GWh) |
|---------|------------------------------|-------------------------------------|----------------------------------|
| unknown | 468                          |                                     | 33100                            |
| solar   | 45                           | 159                                 | 8144                             |
| wind    | 11                           | 9842                                | 15339                            |


We get (I'm linking the csv used for the computation)

[new_unknown_factor_NL.csv](https://github.com/tmrowco/electricitymap-contrib/files/6891824/new_unknown_factor_NL.csv)


__342 gCO2eq/kWh__


@FelixDQ 🟢 for updating the parser 

Resolving patch:
diff --git a/parsers/NL.py b/parsers/NL.py
--- a/parsers/NL.py
+++ b/parsers/NL.py
@@ -12,7 +12,7 @@
 
 
 def fetch_production(zone_key='NL', session=None, target_datetime=None,
-                     logger=logging.getLogger(__name__), energieopwek_nl=True):
+                     logger=logging.getLogger(__name__), energieopwek_nl=False):
     if target_datetime is None:
         target_datetime = arrow.utcnow()
     else:


Test patch: