Repository: mampfes/hacs_waste_collection_schedule

Issue:
[Bug]: source 'City of Karlsruhe' stopped working
### I Have A Problem With:

A specific source

### What's Your Problem

Release 1.44.0:
Due to changes on the website the source '**City of Karlsruhe**' (name: karlsruhe_de) stopped working.
I start troubleshooting and add my findings here.

### Source (if relevant)

karlsruhe_de

### Logs

_No response_

### Relevant Configuration

_No response_

### Checklist Source Error

- [x] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
- [x] Checked that the website of your service provider is still working
- [x] Tested my attributes on the service provider website (if possible)
- [X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

### Checklist Sensor Error

- [x] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

### Required

- [X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
- [X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.


Discussion:
Looks like the API URL https://web6.karlsruhe.de/service/abfall/akal/akal.php changed to https://web6.karlsruhe.de/service/abfall/akal_2023/akal.php.

Change line `with open("test.html", "w") as f:` to `with open("test.html", "w", encoding="utf-8") as f:` ??
Change line `bin_type = row.find("div", class_="col_3-2")` to `bin_type = row.find("div", class_="col_6-2")` ??
Change line `pickup_col = row.find("div", class_="col_3-3")` to `pickup_col = row.find("div", class_="col_6-3")` ??

This code (file: custom_components\waste_collection_schedule\waste_collection_schedule\source\karlsruhe_de.py) works for me:
```
import datetime
import re

import requests
from bs4 import BeautifulSoup
from waste_collection_schedule import Collection  # type: ignore[attr-defined]

TITLE = "City of Karlsruhe"
DESCRIPTION = "Source for City of Karlsruhe."
URL = "https://www.karlsruhe.de/"
TEST_CASES = {
    "Östliche Rheinbrückenstraße 1": {
        "street": "Östliche Rheinbrückenstraße",
        "hnr": 1
    },
    "Habichtweg 4": {
        "street": "Habichtweg", 
        "hnr": 4
    },
    "Machstraße 5": {
        "street": "Machstraße", 
        "hnr": 5
    },
    "Bernsteinstraße 10 ladeort 1": {
        "street": "Bernsteinstraße",
        "hnr": 10,
        "ladeort": 1
    },
    "Bernsteinstraße 10 ladeort 2": {
        "street": "Bernsteinstraße",
        "hnr": 10,
        "ladeort": 2
     }
}


ICON_MAP = {
    "Restmüll": "mdi:trash-can",
    "Bioabfall": "mdi:leaf",
    "Papier": "mdi:package-variant",
    "Wertstoff": "mdi:recycle",
    "Sperrmüllabholung": "mdi:wardrobe"
}


API_URL = "https://web6.karlsruhe.de/service/abfall/akal_2023/akal.php"


class Source:
    def __init__(self, street: str, hnr: str | int, ladeort: int | None = None):
        self._street: str = street
        self._hnr: str | int = hnr
        self._ladeort: int | None = ladeort

    def fetch(self):
        args = {
            "strasse_n": self._street,
            "hausnr": self._hnr,
            "ladeort": self._ladeort,
            "anzeigen": "anzeigen"
        }

        # get json file
        r = requests.post(API_URL, data=args, params={"hausnr=": ""})
        r.raise_for_status()

        with open("test.html", "w", encoding="utf-8") as f:
            f.write(r.text)

        soup = BeautifulSoup(r.text, "html.parser")
        rows = soup.find_all("div", class_="row")
        entries = []

        for row in rows:
            column = row.find("div", class_="col_6-2")
            
            if column is None or not column.contents:
                column = row.find("div", class_="col_7-3")
                if column is None or not column.contents:
                    continue

                for content in column.contents:
                    if content.text.startswith("Sperrmüllabholung"):
                        bin_type = column.contents[0].text.strip()
                        if bin_type.endswith(":"):
                            bin_type = bin_type[:-1].strip()
                        icon = ICON_MAP.get(bin_type)  # Collection icon

                    elif content.text.startswith("Straßensperrmüll"):
                        dates = re.findall(r"\d{2}\.\d{2}\.\d{4}", content.text)
                        date = datetime.datetime.strptime(dates[0], "%d.%m.%Y").date()

                entries.append(Collection(date=date, t=bin_type, icon=icon))

            else:
                bin_type = column.contents[0].text.split(",")[0].strip()
                icon = ICON_MAP.get(bin_type)  # Collection icon

                pickup_col = row.find("div", class_="col_6-3")
                if pickup_col is None or not pickup_col.contents:
                    continue

                for date in re.findall(r"\d{2}\.\d{2}\.\d{4}", pickup_col.text):
                    date = datetime.datetime.strptime(date, "%d.%m.%Y").date()
                    entries.append(Collection(date=date, t=bin_type, icon=icon))

        return entries
```
The code works for me, too. I use API_URL https://web6.karlsruhe.de/service/abfall/akal/akal_2024.php instead. Result is the same.

Thanks!
According to the main website https://tsk.karlsruhe.de/abfuhrkalender, the API URL is now:
https://web6.karlsruhe.de/service/abfall/akal/akal_2024.php
fixed in #1657 
This will be part of release 1.45.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)
@5ila5 
> fixed in #1657 This will be part of release 1.45.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)


This fix did not work for me (or it broke again?).
The request always responds with 404.

I had to do following changes to finally make it work, additionally to the fix in release 1.45.0:

- Change the API_URL to `https://web4.karlsruhe.de/...` instead of `https://web6.karlsruhe.de/...`
- AND send the request (line 68) with an additional `verify=False` argument

Seems like they changed the Api Url again. Their website now also points to web4

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/karlsruhe_de.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/karlsruhe_de.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/karlsruhe_de.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/karlsruhe_de.py
@@ -1,9 +1,17 @@
 from datetime import datetime
 
 import requests
+import urllib3
 from waste_collection_schedule import Collection  # type: ignore[attr-defined]
 from waste_collection_schedule.service.ICS import ICS
 
+# With verify=True the POST fails due to a SSLCertVerificationError.
+# Using verify=False works, but is not ideal. The following links may provide a better way of dealing with this:
+# https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
+# https://urllib3.readthedocs.io/en/1.26.x/user-guide.html#ssl
+# These two lines areused to suppress the InsecureRequestWarning when using verify=False
+urllib3.disable_warnings()
+
 TITLE = "City of Karlsruhe"
 DESCRIPTION = "Source for City of Karlsruhe."
 URL = "https://www.karlsruhe.de/"
@@ -36,7 +44,7 @@
 }
 
 
-API_URL = "https://web6.karlsruhe.de/service/abfall/akal/akal_{year}.php"
+API_URL = "https://web{i}.karlsruhe.de/service/abfall/akal/akal_{year}.php"
 
 
 class Source:
@@ -50,10 +58,11 @@ def fetch(self):
         now = datetime.now()
         error = None
         for year in (now.year, now.year + 1, now.year - 1):
-            try:
-                return self.get_data(API_URL.format(year=year))
-            except Exception as e:
-                error = e
+            for i in (4, 6):
+                try:
+                    return self.get_data(API_URL.format(year=year, i=i))
+                except Exception as e:
+                    error = e
         raise error
 
     def get_data(self, url):
@@ -65,7 +74,7 @@ def get_data(self, url):
         }
         params = {"hausnr": self._hnr}
 
-        r = requests.post(url, data=data, params=params)
+        r = requests.post(url, data=data, params=params, verify=False)
         dates = self.ics.convert(r.text)
 
         entries = []


Test patch: