Repository: electricitymaps/electricitymaps-contrib

Issue:
CA-SK production parser down
## Description

This is an automatic error report generated for Canada Saskatchewan (CA-SK).

Issues:
- No recent data found for `production` parser
- No recent data found for `consumption` parser

## Suggestions
- Try running the parser locally using the command `poetry run test_parser CA-SK production`
- <a href="https://storage.googleapis.com/electricitymap-parser-logs/CA-SK.html">Explore the runtime logs</a>

You can see an overview of all parser issues [here](https://github.com/tmrowco/electricitymap-contrib/wiki/Parser-issues).



Discussion:
This is logging an SSL certificate error:

`HTTPSConnectionPool(host='www.saskpower.com', port=443): Max retries exceeded with url: /ignitionapi/Content/GetNetLoad (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))`

The certificate looks fine to me. www.saskpower.com loads fine for me (Firefox 113.0.2/Linux), cert shows as valid until September 2023, issued by entrust.net.

The consumption URL https://www.saskpower.com/ignitionapi/Content/GetNetLoad is returning a `"Data Unavailable"` message.

However, the production URL https://www.saskpower.com/ignitionapi/PowerUseDashboard/GetPowerUseDashboardData does return data as expected.

Is anyone able to see if there's any SSL cert weirdness on the crawler machines?
Possibly just disable SSL verification, as in https://github.com/electricitymaps/electricitymaps-contrib/pull/5556 ?
If you want feel free to open a PR for it otherwise I'll do so tomorrow. 
I can take a look.
Looks like an incomplete certificate chain: https://www.ssllabs.com/ssltest/analyze.html?d=www.saskpower.com
See: https://stackoverflow.com/questions/46604114/python-requests-ssl-error-certificate-verify-failed
We need to add the intermediary CA cert to the `cacert.pem` file in `.venv/lib/python3.8/site-packages/certifi/cacert.pem`, or somehow include another one in the function call, although I'm not sure how to specify a relative file path or where to put a new `pem` file. @VIKTORVAV99 any ideas?

Some sample code from the SO post:
```
requests.get("https://...", verify='my_trust_store.pem')
```
Certificate in question:

-----BEGIN CERTIFICATE-----
MIIFDjCCA/agAwIBAgIMDulMwwAAAABR03eFMA0GCSqGSIb3DQEBCwUAMIG+MQswCQYDVQQGEwJV
UzEWMBQGA1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2VlIHd3dy5lbnRydXN0Lm5ldC9s
ZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMDkgRW50cnVzdCwgSW5jLiAtIGZvciBhdXRob3Jp
emVkIHVzZSBvbmx5MTIwMAYDVQQDEylFbnRydXN0IFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
dHkgLSBHMjAeFw0xNTEwMDUxOTEzNTZaFw0zMDEyMDUxOTQzNTZaMIG6MQswCQYDVQQGEwJVUzEW
MBQGA1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2VlIHd3dy5lbnRydXN0Lm5ldC9sZWdh
bC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMTIgRW50cnVzdCwgSW5jLiAtIGZvciBhdXRob3JpemVk
IHVzZSBvbmx5MS4wLAYDVQQDEyVFbnRydXN0IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gTDFL
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2j+W0E25L0Tn2zlem1DuXKVh2kFnUwmq
AJqOV38pa9vH4SEkqjrQjUcj0u1yFvCRIdJdt7hLqIOPt5EyaM/OJZMssn2XyP7BtBe6CZ4DkJN7
fEmDImiKm95HwzGYei59QAvS7z7Tsoyqj0ip/wDoKVgG97aTWpRzJiatWA7lQrjV6nN5ZGhTJbiE
z5R6rgZFDKNrTdDGvuoYpDbwkrK6HIiPOlJ/915tgxyd8B/lw9bdpXiSPbBtLOrJz5RBGXFEaLpH
PATpXbo+8DX3Fbae8i4VHj9HyMg4p3NFXU2wO7GOFyk36t0FASK7lDYqjVs1/lMZLwhGwSqzGmId
TivZGwIDAQABo4IBDDCCAQgwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwMwYI
KwYBBQUHAQEEJzAlMCMGCCsGAQUFBzABhhdodHRwOi8vb2NzcC5lbnRydXN0Lm5ldDAwBgNVHR8E
KTAnMCWgI6Ahhh9odHRwOi8vY3JsLmVudHJ1c3QubmV0L2cyY2EuY3JsMDsGA1UdIAQ0MDIwMAYE
VR0gADAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTAdBgNVHQ4EFgQU
gqJwdN28Uz/Pe9T3zX+nYMYKTL8wHwYDVR0jBBgwFoAUanImetAe733nO2lR1GyNn5ASZqswDQYJ
KoZIhvcNAQELBQADggEBADnVjpiDYcgsY9NwHRkwy/YJrMxp1cncN0HyMg/vdMNY9ngnCTQIlZIv
19+4o/0OgemknNM/TWgrFTEKFcxSBJPok1DD2bHi4Wi3Ogl08TRYCj93mEC45mj/XeTIRsXsgdfJ
ghhcg85x2Ly/rJkCk9uUmITSnKa1/ly78EqvIazCP0kkZ9Yujs+szGQVGHLlbHfTUqi53Y2sAEo1
GdRvc6N172tkw+CNgxKhiucOhk3YtCAbvmqljEtoZuMrx1gL+1YQ1JH7HdMxWBCMRON1exCdtTix
9qrKgWRs6PLigVWXUX/hwidQosk8WwBD9lu51aX8/wdQQGcHsFXwt35uLcw=
-----END CERTIFICATE-----
I think it's just easier to skip verification entirely with verify=false in the get request rather than start messing around with certs. 
Won't that introduce security vulnerabilities?
While the connection would get less secure it don't introduce any vulnerabilities per say. As we are only reading data the only risk would be a man in the middle attack that would replace it with falsified data. If they don't replace the data the parser would simply crash.

The reason I think this might be better is because if we manually add the public key that is missing and they later update it we would have to update that as well which causes extra maintenance.

I would also like to note we can't update the certificates inside python packages as these would be overwritten on each poetry install step (which I believe happens on every deployment).

Resolving patch:
diff --git a/parsers/CA_SK.py b/parsers/CA_SK.py
--- a/parsers/CA_SK.py
+++ b/parsers/CA_SK.py
@@ -115,7 +115,7 @@ def fetch_consumption(
     # Set the headers to mimic a user browser as the API will return a 403 if not.
     headers = {"user-agent": USER_AGENT}
 
-    response: Response = session.get(CONSUMPTION_URL)  # , headers=headers)
+    response: Response = session.get(CONSUMPTION_URL, headers=headers)
 
     if not response.ok:
         raise ParserException(


Test patch: