Repository: mampfes/hacs_waste_collection_schedule

Issue:
ICS Calendar not working anymore, last year there were no problems (fetch failed)
Hey everyone and a happy new year,

I´ve been using this Integration for the last year without Issues. But I can not get it to work for 2023...

This is one of the URLs for my City:

https://servicebetrieb.koblenz.de/abfallwirtschaft/entsorgungstermine-digital/entsorgungstermine-2023-digital/altstadt-2023.ics?cid=2ui7

My configuration.yaml looks like this:

waste_collection_schedule:
  sources:
    - name: ics
      args:
        url: "https://servicebetrieb.koblenz.de/abfallwirtschaft/entsorgungstermine-digital/entsorgungstermine-2023-digital/altstadt-2023.ics?cid=2ui7"
        version: 1
      customize:
        - type: Altpapier       
          alias: Papier
        - type: Gelber Sack       
          alias: Gelber Sack   

I´ve tried to use verision 1 or 2, opened the calendar with my PC - reinstalled the Integration... I use HA-Core as a Docker Image

The Error is: 

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
Integration: waste_collection_schedule (documentation)
First occurred: 14:46:42 (1 occurrences)
Last logged: 14:46:42

fetch failed for source ICS: Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/icalendar/parser.py", line 340, in parts validate_token(name) File "/usr/local/lib/python3.10/site-packages/icalendar/parser.py", line 128, in validate_token raise ValueError(name) ValueError: ﻿BEGIN During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py", line 181, in fetch return self.fetch_url(self._url, self._params) File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py", line 202, in fetch_url return self._convert(r.text) File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py", line 209, in _convert dates = self._ics.convert(data) File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/ICS_v1.py", line 21, in convert calendar = icalendar.Calendar.from_ical(ics_data) File "/usr/local/lib/python3.10/site-packages/icalendar/cal.py", line 330, in from_ical name, params, vals = line.parts() File "/usr/local/lib/python3.10/site-packages/icalendar/parser.py", line 352, in parts raise ValueError( ValueError: Content line could not be parsed into parts: '﻿BEGIN:VCALENDAR': ﻿BEGIN


Any Ideas what I did wrong? I just thought I´d switch out URLs and be done.

Thx


Discussion:
Have the same issue. Imported the ics manually. Last year all was working. The Events also showing in the HA calendar but the sensors from the waste_collection_schedule addon cant get the date.


@TheDeath24 

For me the Events do not show up in the Calendar. So my guess is the Addon cannot even read the ics File. I just don´t know why.
The ICS file contains only dates from the past year. And obviously it is malformed. Probably this is caused by the year transition and will be hopefully fix itself once they have updated the dates. Please check again in a few days and let me know.
> @TheDeath24
> 
> For me the Events do not show up in the Calendar. So my guess is the Addon cannot even read the ics File. I just don´t know why.

You are right, then we have a different problem. My Dates are in the calendar but the sensors are not getting the dates.
@TheDeath24 If the dates are visible in the calendar, but the sensors are unavailable then this is typically are problem with the `types` argument in the sensor configuration. The names below `types` has to match the original waste type exactly!
Any updates on this?
@mampfes 

I´ve tried a few times to get it to work, but no success.

When Downloading and opening the file with Windows it opens with the Calendar App and the dates show up... 


I think the issue is with the length of the `DESCRIPTION` lines.
[iCalendar Validator](https://icalendar.org/validator.html?url=https://servicebetrieb.koblenz.de/abfallwirtschaft/entsorgungstermine-digital/entsorgungstermine-2023-digital/altstadt-2023.ics?cid=2ui7) flags a number of warnings about line length. If you shorten those, and rerun the validation it flags more long lines - I guess the page is limited to displaying 10 messages.

If all the `DESCRIPTION` lines are reduced to <70 characters and the config set up to use that edited file rather than the url then the collections appear in the calendar.

```yaml
sources:
    - name: ics
      args:
        file: "./MyStuff/test.ics"
        # url: "https://servicebetrieb.koblenz.de/abfallwirtschaft/entsorgungstermine-digital/entsorgungstermine-2023-digital/altstadt-2023.ics?cid=2ui7"
        version: 1
```
![image](https://user-images.githubusercontent.com/86194065/213885407-732152f5-8541-48af-99f3-8c2634cb669c.png)


I guess python's processing of ics files doesn't handle long description very well, or the source provider needs to come up with a more concise `DESCRIPTION`. 

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py
@@ -115,6 +115,9 @@
             "download": "ical",
         },
     },
+    "UTF-8-SIG (UTF-8 with BOM)": {
+        "url": "https://servicebetrieb.koblenz.de/abfallwirtschaft/entsorgungstermine-digital/entsorgungstermine-2023-digital/altstadt-2023.ics?cid=2ui7",
+    },
 }
 
 
@@ -202,7 +205,11 @@ def fetch_url(self, url, params=None):
             )
         r.raise_for_status()
 
-        r.encoding = "utf-8"  # requests doesn't guess the encoding correctly
+        if r.apparent_encoding == "UTF-8-SIG":
+            r.encoding = "UTF-8-SIG"
+        else:
+            r.encoding = "utf-8"  # requests doesn't guess the encoding correctly
+
         return self._convert(r.text)
 
     def fetch_file(self, file):


Test patch: