Repository: mampfes/hacs_waste_collection_schedule

Issue:
[Bug]: No Dates being pulled anymore - awido_de
### I Have A Problem With:

A specific source

### What's Your Problem

my configuration has worked fined before (without housenumber), but since a few days or weeks there are no more data.

I have tried to add the housenumber, but the result is the same.


waste_collection_schedule:
  sources:
    - name: awido_de
      args:
        customer: aic-fdb
        city: Kissing
        street: XXXstraße
      customize:
        - type: Restmülltonne
          alias: Restmülltonne
          icon: mdi:trash-can

### Source (if relevant)

awido_de , customer: aic-fdb

### Logs

```
This error originated from a custom integration.

Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
Integration: waste_collection_schedule (documentation)
First occurred: 07:46:24 (1 occurrences)
Last logged: 07:46:24

fetch failed for source AWIDO Online: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/simplejson/__init__.py", line 514, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/simplejson/decoder.py", line 386, in decode obj, end = self.raw_decode(s) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/simplejson/decoder.py", line 416, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/awido_de.py", line 342, in fetch cal_json = r.json() ^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```


### Relevant Configuration

```YAML
waste_collection_schedule:
  sources:
    - name: awido_de
      args:
        customer: aic-fdb
        city: Kissing
        street: XXXstraße
      customize:
        - type: Restmülltonne
          alias: Restmülltonne
          icon: mdi:trash-can
```


### Checklist Source Error

- [X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
- [X] Checked that the website of your service provider is still working
- [ ] Tested my attributes on the service provider website (if possible)
- [X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

### Checklist Sensor Error

- [X] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

### Required

- [X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
- [X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.


Discussion:
When I use one of the example data I get values (at leas for Biotonne and Gelbe Tonne):

    - name: awido_de
      args:
        customer: lra-dah
        city: Altomünster
        street: Maisbrunn
Same issue here.
Same customer, other city.
Error log is identical.

Same issue here but let me enhance information. Root cause seems to be that awido delivers empty response with http 200 for getData. As this is a simple get request, the issue can be reproduced e.g. in the following way:

https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getPlaces/client=aic-fdb
-->returns places and their ids

In the next line the id from "Affing" is used, which is just a sample ;-)
https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getGroupedStreets/8bd3a1cc-ef7c-4413-a20e-7fea9e3e314b?client=aic-fdb
-->returns valid streets and their ids

In the next line, I used the id from "Am Hang":
[https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getData/<valid_street_id>?client=aic-fdb](https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getData/590ef8cf-169f-43e9-a27c-44222ece3d7a?client=aic-fdb)
-->returns with http 200 but without content
-->If you change the id to an invalid one you retrieve "Invalid Ortsteil" which let me think that the webservice is working in general
-->If you change getPlaces e.g. to the id of "Unterhaching" you retrieve values.

All in all I am wondering if this is a temporary issue with missing data or if getData is no longer working in the same way as before and that this is a one by one release from awido. 

In addtion, the calendar.aspx is still working:
https://awido.cubefour.de/Customer/aic-fdb/v2/Calendar2.aspx

Here is the working example for Unterhaching:
https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getGroupedStreets/00000000-0000-0000-1000-000000000001?client=unterhaching
-->selected id of "Am Tränkeanger"
https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getData/f5a9a5a8-d42d-4ccd-b26b-c603510f0149?client=unterhaching
--> returns expected information


Hope that helps to narrow the topic if this is related to a new version of the api. Otherwise this is nothing that can be changed within the integration.


Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/awido_de.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/awido_de.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/awido_de.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/awido_de.py
@@ -3,6 +3,7 @@
 
 import requests
 from waste_collection_schedule import Collection  # type: ignore[attr-defined]
+from waste_collection_schedule.service.ICS import ICS
 
 TITLE = "AWIDO Online"
 DESCRIPTION = "Source for AWIDO waste collection."
@@ -255,17 +256,27 @@ def EXTRA_INFO():
         "city": "Spalt",
         "street": "Pflugsmühler Weg",
     },
+    "Kissing Karwendelweg": {
+        "customer": "aic-fdb",
+        "city": "Kissing",
+        "street": "Karwendelweg",
+    },
 }
 
 _LOGGER = logging.getLogger(__name__)
 
 
+class JSONNotSupported(Exception):
+    pass
+
+
 class Source:
     def __init__(self, customer, city, street=None, housenumber=None):
         self._customer = customer
         self._city = city
         self._street = street
         self._housenumber = None if housenumber is None else str(housenumber)
+        self._ics = ICS()
 
     def fetch(self):
         # Retrieve list of places
@@ -333,12 +344,48 @@ def fetch(self):
 
                 oid = hsnbr_to_oid[self._housenumber]
 
+        try:
+            return self.get_json_data(oid)
+        except JSONNotSupported:
+            return self.get_ics_data(oid)
+
+    def get_ics_data(self, oid) -> list[Collection]:
+        now = datetime.datetime.now()
+
+        entries: list[Collection] = []
+
+        for year in [now.year, now.year + 1] if now.month == 12 else [now.year]:
+            r = requests.get(
+                f"https://awido.cubefour.de/Customer/{self._customer}/KalenderICS.aspx",
+                params={
+                    "oid": oid,
+                    "jahr": year,
+                    "fraktionen": "",
+                    "reminder": "-1.17:00",
+                },
+            )
+            r.raise_for_status()
+            ics_file = r.text
+
+            dates = self._ics.convert(ics_file)
+
+            entries = []
+            for d in dates:
+                # prevents duplicates
+                if any(e.date == d[0] and e.type == d[1] for e in entries):
+                    continue
+                entries.append(Collection(d[0], d[1]))
+
+        return entries
+
+    def get_json_data(self, oid) -> list[Collection]:
         # get calendar data
         r = requests.get(
             f"https://awido.cubefour.de/WebServices/Awido.Service.svc/secure/getData/{oid}",
             params={"fractions": "", "client": self._customer},
         )
-        r.raise_for_status()
+        if r.status_code != 200 or r.text.strip() == "":
+            raise JSONNotSupported()
         cal_json = r.json()
 
         # map fraction code to fraction name


Test patch: