Repository: mampfes/hacs_waste_collection_schedule

Issue:
[Bug]: heilbronn _de isn’t working anymore
### I Have A Problem With:

A specific source

### What's Your Problem

It looks like Heilbronn is switching they’re service. The current heilbronn_de isn’t working anymore.

The new service is published under https://abfallwirtschaft.heilbronn.de/




### Source (if relevant)

_No response_

### Logs

_No response_

### Relevant Configuration

_No response_

### Checklist Source Error

- [ ] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
- [ ] Checked that the website of your service provider is still working
- [ ] Tested my attributes on the service provider website (if possible)
- [X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

### Checklist Sensor Error

- [ ] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

### Required

- [X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
- [X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.


Discussion:
The old Heilbronn source only gets dates till 2024-01-31. 

I tried to take a look at their new calendar, but I could not figure out how to get the dates:

As far as I could figure out they just send requests to https://api.heilbronn.de/garbage-calendar?method=get&datatype=districts which does contain all the addresses and its available waste types, but I could not find any request getting any actual dates, so I guess it's probably part of the JavaScript, but the JavaScript is nearly unreadable as it is "compiled" from dart/flutter. The App does exactly the same: 
![grafik](https://github.com/mampfes/hacs_waste_collection_schedule/assets/38183212/bdb3a3c2-a048-4df5-8498-ff8ac81be231)

So I currently do not see a way to get this source up and running again. You could ask them if there is a way to get an ICS/ICAL file to use with the [ICS source](https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/doc/source/ics.md), but they currently do not support it on their website.

You may be able to use the app to import the events in your calendar (download dropdown only available in the mobile app) and then export them to an ICS file, upload to your HA instance and use the ICS source but this far from ideal.

Or you can use the [User Specified/static source](https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/doc/source/static.md) and define your dates "by hand".
The dates are here https://api.heilbronn.de/garbage-calendar?method=get&datatype=pickupdates

Workflow would be
1. Load https://api.heilbronn.de/garbage-calendar?method=get&datatype=districts
2. Find address (sometime * is used for the complete street, example "74072,Achtungstrasse")
3. Take the collection-calendar id by bin type
4. Load https://api.heilbronn.de/garbage-calendar?method=get&datatype=pickupdates
5. Select the calender based on the data from step 3
Thanks @ReneNulschDE and @5ila5 for all you effort.
I tried to build a new script for the new API. When i run the code on my local machine i get a nice collection with data. But it wont work if i copy over the script to home assistant to use it there. Maybe one of you can do a quick check of my code?

```python
from datetime import datetime

import requests
from waste_collection_schedule import Collection  # type: ignore[attr-defined]
from waste_collection_schedule.service.ICS import ICS

TITLE = "Heilbronn Entsorgungsbetriebe"
DESCRIPTION = "Source for city of Heilbronn, Germany."
URL = "https://abfallwirtschaft.heilbronn.de/"
TEST_CASES = {
    "Rosenau": {
        "plz": 74072,
        "strasse": "Rosenau",
    },
    "Biberach": {
        "strasse": "Kehrhüttenstraße",
        "plz": 74078,
    },
    "Klingenberg:": {
        "strasse": "Wittumhalde",
        "plz": "74081",
    },
    "Klingenberg (hausnr as string):": {
        "strasse": "Wittumhalde",
        "plz": "74081",
    },
    "Rosenbergstraße 53": {
        "strasse": "Rosenbergstraße",
        "plz": "74074",
    },
    "Rosenbergstraße 41": {
        "strasse": "Rosenbergstraße",
        "plz": "74072",
    },
}

ICON_MAP = {
    "r": "mdi:trash-can",
    "b": "mdi:leaf",
    "c": "mdi:leaf",
    "l": "mdi:recycle",
    "a": "mdi:package-variant",
    "p": "mdi:package-variant",
}



class Source:
    def __init__(self, plz: int, strasse: str):
        self._plz: str = str(plz)
        self._strasse: str = strasse
        self._ics = ICS()

    def fetch(self):
        r = requests.get(
            "https://api.heilbronn.de/garbage-calendar?method=get&datatype=districts"
        )
        r.raise_for_status()
        data = r.json()

        districts = data["data"][self._plz][self._strasse]["*"]

        # filter waste type
        filteredData = {
            "light-packaging": districts.get("yellow"),
            "paper": districts.get("paper"),
            "bio_2": districts.get("bio"),
            "bio_1": districts.get("bio"),
            "christmastree": districts.get("green"),
            "residual_2": districts.get("residual")
        }

        r = requests.get(
                    "https://api.heilbronn.de/garbage-calendar?method=get&datatype=pickupdates"
                )
        r.raise_for_status()
        pickupDates = r.json()

        entries = []

        for keyDistrict, valueDistrict in filteredData.items():
            value = pickupDates["data"][valueDistrict][keyDistrict]
            for key2, value2 in value.items():
                date: dict = datetime.strptime(value2, "%Y-%m-%d").date()
                entry = keyDistrict
                if entry in ICON_MAP:
                    icon = ICON_MAP.get(entry)
                else:
                    icon = ICON_MAP.get(entry[0])
                entries.append(Collection(date, entry, icon))

                return entries

```
The indentation of the line "return entries" is wrong. In the current structure you return after the first entry. Bring it to the same level like "for keyD...."

There are multiple things that you need to check:
1. A house number is needed for some streets
2. There more bin types
3. You can filter out old entries. The dateset has entries from the past.
3. point @ReneNulschDE is not valid 

> - A source script should return all data for all available waste types.
> - A source script should not provide options to limit the returned waste types.
> - A source script should return all data for the entire time period available (including past dates if they are returned).
> - A source script should not provide a configuration option to limit the requested time frame.

https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/doc/contributing_source.md#python-source-script



did some quick changes, needs testing: ([how to test](https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/doc/contributing_source.md#test-the-new-source-file))

```python
from datetime import datetime

import requests
from waste_collection_schedule import Collection  # type: ignore[attr-defined]
from waste_collection_schedule.service.ICS import ICS

TITLE = "Heilbronn Entsorgungsbetriebe"
DESCRIPTION = "Source for city of Heilbronn, Germany."
URL = "https://heilbronn.de"
TEST_CASES = {
    "Klingenberg (hausnr as string):": {
        "strasse": "Wittumhalde",
        "plz": "74081",
        # "hausnr": "75",
    },
    "Rosenbergstraße 53": {
        "strasse": "Rosenbergstraße",
        "plz": "74074",
        "hausnr": "53",
    },
    "Rosenbergstraße 41": {
        "strasse": "Rosenbergstraße",
        "plz": "74072",
        "hausnr": "41",
    },
}

ICON_MAP = {
    "R": "mdi:trash-can",
    "B": "mdi:leaf",
    "GRÜN": "mdi:leaf",
    "G": "mdi:recycle",
    "A": "mdi:package-variant",
    "BT": "mdi:package-variant",
}


class Source:
    def __init__(self, plz: int, strasse: str, hausnr: str | int| None = None):
        self._plz: str = str(plz)
        self._strasse: str = strasse
        self._hausnr: str = str(hausnr) if hausnr is not None else None

    def fetch(self):
        r = requests.get(
            "https://api.heilbronn.de/garbage-calendar?method=get&datatype=districts"
        )
        r.raise_for_status()
        data = r.json()

        street = data["data"][self._plz][self._strasse]
        if not self._hausnr or "*" in street:
            districts:dict = street["*"]
        else:
            districts:dict = street[self._hausnr]

        # filter waste type
        collection_keys = [value for key,value in districts.items() if key not in ("city", "district")]

        r = requests.get(
                    "https://api.heilbronn.de/garbage-calendar?method=get&datatype=pickupdates"
                )
        r.raise_for_status()
        pickupDates = r.json()

        entries = []

        for valueDistrict in collection_keys:
            value = pickupDates["data"][valueDistrict]
            for collection_type, collection_dates in value.items():
                for value2 in collection_dates.values():
                    date: dict = datetime.strptime(value2, "%Y-%m-%d").date()
                    entry = collection_type
                    if entry in ICON_MAP:
                        icon = ICON_MAP.get(entry)
                    else:
                        icon = ICON_MAP.get(entry[0])
                    entries.append(Collection(date, entry, icon))

        return entries
```
Thank you for your adjustments. It works nice, but the mapping for the icons are not right anymore. The trash type is now in english. Maybe you can take my mapping:

```python
ICON_MAP = {
    "r": "mdi:trash-can",
    "b": "mdi:leaf",
    "c": "mdi:leaf",
    "g": "mdi:leaf",
    "l": "mdi:recycle",
    "a": "mdi:package-variant",
    "p": "mdi:package-variant",
}
```
Was muss ich tun, um jetzt wieder Benachrichtigungen via Home Assistant erhalten zu können? In der Doku hat sich noch nichts geändert. 
Is this new source already merged or is there already a pr?

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/heilbronn_de.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/heilbronn_de.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/heilbronn_de.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/heilbronn_de.py
@@ -2,7 +2,6 @@
 
 import requests
 from waste_collection_schedule import Collection  # type: ignore[attr-defined]
-from waste_collection_schedule.service.ICS import ICS
 
 TITLE = "Heilbronn Entsorgungsbetriebe"
 DESCRIPTION = "Source for city of Heilbronn, Germany."
@@ -31,7 +30,7 @@
     "Rosenbergstraße 53": {
         "strasse": "Rosenbergstraße",
         "plz": "74074",
-        "hausnr": "53",
+        "hausnr": "50",
     },
     "Rosenbergstraße 41": {
         "strasse": "Rosenbergstraße",
@@ -41,90 +40,62 @@
 }
 
 ICON_MAP = {
-    "R": "mdi:trash-can",
-    "B": "mdi:leaf",
-    "GRÜN": "mdi:leaf",
-    "G": "mdi:recycle",
-    "A": "mdi:package-variant",
-    "BT": "mdi:package-variant",
+    "residual": "mdi:trash-can",
+    "bio": "mdi:leaf",
+    "green": "mdi:leaf",
+    "light-packaging": "mdi:recycle",
+    "paper": "mdi:package-variant",
+    "paper-bundle": "mdi:package-variant",
+    "christmastree": "mdi:pine-tree",
 }
 
 
 class Source:
-    def __init__(self, plz: int, strasse: str, hausnr: str | int):
+    def __init__(self, plz: int, strasse: str, hausnr: str | int | None = None):
         self._plz: str = str(plz)
         self._strasse: str = strasse
-        self._hausnr: str = str(hausnr)
-        self._ics = ICS()
+        self._hausnr: str | None = str(hausnr) if hausnr else None
 
     def fetch(self):
         r = requests.get(
-            "https://abfallratgeber.heilbronn.de/excel_hn/htdocs/street.json"
+            "https://api.heilbronn.de/garbage-calendar?method=get&datatype=districts"
         )
         r.raise_for_status()
-        streets: dict = r.json()
+        data = r.json()
+
+        street = data["data"][self._plz][self._strasse]
+        if not self._hausnr or "*" in street:
+            if "*" not in street:
+                raise ValueError(
+                    f"Street {self._strasse} needs to be configured with a house number, available house numbers: {list(street.keys())}"
+                )
+            districts: dict = street["*"]
+        else:
+            if self._hausnr not in street:
+                raise ValueError(
+                    f"House number {self._hausnr} not found for street {self._strasse}, available house numbers: {list(street.keys())}"
+                )
+            districts: dict = street[self._hausnr]
+
+        # filter waste type
+        collection_keys = [
+            value for key, value in districts.items() if key not in ("city", "district")
+        ]
 
-        # all the data is spilt in to two files for different kinds of trash
-        year_response = requests.get(
-            "https://abfallratgeber.heilbronn.de/excel_hn/htdocs/yearObject.json"
-        )
-        year_response.raise_for_status()
-
-        yearB_response = requests.get(
-            "https://abfallratgeber.heilbronn.de/excel_hn/htdocs/yearBObject.json"
+        r = requests.get(
+            "https://api.heilbronn.de/garbage-calendar?method=get&datatype=pickupdates"
         )
-        yearB_response.raise_for_status()
-
-        days_list: list[dict] = [year_response.json(), yearB_response.json()]
-
-        keys = []
-
-        # get the values to search for in the json and save them in keys
-        for street in streets:
-
-            if (street["route"] == self._strasse) and (
-                street["postal_code"] == self._plz
-            ):
-
-                if ("street_number" in street) and (
-                    self._hausnr not in street["street_number"]
-                ):
-                    continue
-                keys = []
-                for key in list({street["area"], street["g"], street["bt"]}):
-                    keys.append(key)
-                    if "-" in key:
-                        keys.append(key.split("-")[0])
-                break
-
-        if keys == []:
-            raise Exception(
-                "no address found for "
-                + self._strasse
-                + " "
-                + self._hausnr
-                + " "
-                + self._plz
-            )
+        r.raise_for_status()
+        pickupDates = r.json()
 
         entries = []
-        # iterate over both json files and search for the keys
-        for days in days_list:
-            for day in days.values():
-                if "date" not in day:
-                    continue
-
-                date: dict = datetime.strptime(day["date"], "%d.%m.%Y").date()
-
-                for key in keys:
-                    if key in day["districts"] and day["districts"][key] != []:
-                        for entry in day["districts"][key]:
-                            if entry in ICON_MAP:
-                                icon = ICON_MAP.get(entry)
-                            elif len(entry) >= 2 and entry[:2] == "BT":
-                                icon = ICON_MAP.get("BT")
-                            else:
-                                icon = ICON_MAP.get(entry[0])
-                            entries.append(Collection(date, entry, icon))
 
+        for valueDistrict in collection_keys:
+            value = pickupDates["data"][valueDistrict]
+            for collection_type, collection_dates in value.items():
+                for value2 in collection_dates.values():
+                    date: dict = datetime.strptime(value2, "%Y-%m-%d").date()
+                    entry = collection_type
+                    icon = ICON_MAP.get(entry.split("_")[0].lower())
+                    entries.append(Collection(date, entry, icon))
         return entries


Test patch: