Repository: mampfes/hacs_waste_collection_schedule

Issue:
Berlin Recycling broken due to SSL misconfiguration
Berlin Recycling is currently broken and fails with an `SSLCertVerificationError`:

> Logger: waste_collection_schedule.scraper
> Source: custom_components/waste_collection_schedule/waste_collection_schedule/scraper.py:118
> Integration: waste_collection_schedule (documentation)
> First occurred: 22. März 2022, 16:39:19 (2 occurrences)
> Last logged: 01:12:00
> 
> fetch failed for source Berline Recycling: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn conn.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect self.sock = ssl_wrap_socket( File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kundenportal.berlin-recycling.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/scraper.py", line 116, in fetch entries = self._source.fetch() File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/berlin_recycling_de.py", line 48, in fetch r = session.get(SERVICE_URL, allow_redirects=False) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 517, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='kundenportal.berlin-recycling.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Seems the reason is https://kundenportal.berlin-recycling.de has a misconfigured HTTPS server [which is not sending the intermediate certificate](https://www.ssllabs.com/ssltest/analyze.html?d=kundenportal.berlin-recycling.de) ([Starfield Secure Certificate Authority - G2](https://certs.starfieldtech.com/repository/)). Unlike browsers, python [does not support](https://bugs.python.org/issue18617) automatic fetching of missing intermediates.

I think an option to avoid setting `verify = False` on the request would be pinning the CAs and shipping the certificates with the project.


Discussion:
Thanks for the great explanation. Obviously you are the absolut expert in this area (what I am not).

I already had to use `verify=False` for a source in Auckland, Australia. I agree that CA pinning is probably the cleaner solution. But this would mean that we have to renew the certificates from time to time. Right?

Could you prepare a PR?

Thanks.

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/berlin_recycling_de.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/berlin_recycling_de.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/berlin_recycling_de.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/berlin_recycling_de.py
@@ -45,7 +45,7 @@ def fetch(self):
         session = requests.session()
 
         # first get returns session specific url
-        r = session.get(SERVICE_URL, allow_redirects=False)
+        r = session.get(SERVICE_URL, allow_redirects=False, verify=False)
 
         # get session id's
         r = session.get(r.url)


Test patch: