Repository: mampfes/hacs_waste_collection_schedule

Issue:
[Bug]: Haringey Waste Collection sensor returning 'Unknown'
### I Have A Problem With:

A specific source

### What's Your Problem

I am having an issue adding the Haringey council (UK) Waste Collection sensor into HA. The sensor value shows as 'Unknown'.

I have added this code to my configuration.yaml (replacing "My UPRN" with my actual UPRN):

```
  sources:
    - name: haringey_gov_uk
      args:
        uprn: "My UPRN"

sensor:
  - platform: waste_collection_schedule
    name: "Haringey Waste Collection"
    details_format: upcoming
    value_template: 'in {{value.daysTo}} days'
```

![haringeywastecollection](https://github.com/mampfes/hacs_waste_collection_schedule/assets/26800901/41d18b20-8922-48b1-94dc-865ae9ac568c)

I have tested with other configs for other councils I found online and those work as expected. Is there something wrong with the Haringey data feed? The URL mentioned in the documentation is still correct, and I can see the correct information on the Haringey website.

### Source (if relevant)

_No response_

### Logs

_No response_

### Relevant Configuration

_No response_

### Checklist Source Error

- [X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
- [X] Checked that the website of your service provider is still working
- [X] Tested my attributes on the service provider website (if possible)
- [X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

### Checklist Sensor Error

- [X] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

### Required

- [X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
- [X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.


Discussion:
Our test cases work fine UPRNs: 100021209182, 100021207181, 100021202738, 100021202131

You can test your UPRN if you visit "https://wastecollections.haringey.gov.uk/property/ `YOUR_UPRN`" you should see something like this 
![image](https://github.com/mampfes/hacs_waste_collection_schedule/assets/38183212/62a89faf-79e3-453f-8982-1fea8a9e5152)

With specific dates. Did you check your logs?
Hi, yes I have confirmed my UPRN works correctly on the Haringey website.

When you say to check my logs, is there a way I can check logs for the waste collection integration itself? Or just add the details again and see if anything shows up in the logs?
Your first test UPRN worked immediately. My UPRN does not:

![image](https://github.com/mampfes/hacs_waste_collection_schedule/assets/26800901/567de1d1-c443-42ab-8030-890a445f02bc)

I have additional bins at my property listed on the Haringey website, Communal Food and Communal Refuse. Could those be causing issues?
Just add your config -> restart HA -> you may see something in your logs (setting -> system -> logs)

The source takes the normal url and looks for all elements with the class `service-wrapper` in there it looks for an element with class `service-name`, its text will be the name. And for an element td element with class next-service which should contain the next date.

The source may fail if there is a service-wrapper div without a `service-name` and `td.next-service` element or if the `td.next-service` element does not contain a date of format `%d/%m/%Y`
It did throw an error:
```

This error originated from a custom integration.

Logger: custom_components.uk_bin_collection.sensor
Source: helpers/update_coordinator.py:315
integration: UK Bin Collection Data (documentation, issues)
First occurred: 15:30:03 (1 occurrences)
Last logged: 15:30:03

Unexpected error fetching Haringey (TEST) data: 'NoneType' object has no attribute 'find'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 96, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 115, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 78, in template_method
    bin_data_dict = self.parse_data(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/councils/HaringeyCouncil.py", line 37, in parse_data
    .find(text=date_regex)
     ^^^^
AttributeError: 'NoneType' object has no attribute 'find'
```

Which also caused the following error:

```
Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:356
integration: Sensor (documentation, issues)
First occurred: 15:30:04 (1 occurrences)
Last logged: 15:30:04

Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 96, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 115, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 78, in template_method
    bin_data_dict = self.parse_data(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/councils/HaringeyCouncil.py", line 37, in parse_data
    .find(text=date_regex)
     ^^^^
AttributeError: 'NoneType' object has no attribute 'find'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: 'NoneType' object has no attribute 'find'
```

@TomAnks these error messages are not for waste_collection_schedule but for [uk_bin_collection](https://github.com/robbrad/UKBinCollectionData)
@5ila5 My mistake! I had also opened a ticket to report a similar issue with that integration and got the two mixed up.

Here are the error logs for HACS Waste Collection Schedule:

```
Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
integration: waste_collection_schedule ([documentation](https://github.com/mampfes/hacs_waste_collection_schedule#readme))
First occurred: 16:51:52 (1 occurrences)
Last logged: 16:51:52
fetch failed for source Haringey Council: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/haringey_gov_uk.py", line 41, in fetch next_service_date = service_element.select("td.next-service")[0] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^ IndexError: list index out of range
```

As I mentioned before, my UPRN is definitely valid.
@TomAnks Could you please provide the html of the website with your uprn. As our test case work fine (feel free to remove addres or uprn)
I can send you my UPRN URL directly if there's an email I can send it to? Other UPRNs in my area work fine, but mine does not, but would rather not post my UPRN here.
@TomAnks sorry for the misunderstanding, I phrased it really badly, I meant if you could post just the HTML of your page with your uprn/address removed. I currently do not have a public mail address associated with this account
Do you mean like this? https://wastecollections.haringey.gov.uk/property/XXXXXXXXXXXX 

Or do you want the HTML code of the site?
Thank you, I have sent the UPRN and complete URL via email.

On Wed, 17 Apr 2024, 15:55 5ila5, ***@***.***> wrote:

> I'd either need the UPRN (decided you can send it here if you like:
> ***@***.***) or the HTML of the webpage (The script parses the html
> of the https://wastecollections.haringey.gov.uk/property/XXXXXXXXXXXX
> page. So I need the html (either direct or by your UPRN) to check which
> code section needs to be changed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mampfes/hacs_waste_collection_schedule/issues/1952#issuecomment-2061457601>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGMPGBPYVVOYLKL2HEVXQMDY52EO3AVCNFSM6AAAAABF4SWVO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRRGQ2TONRQGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/haringey_gov_uk.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/haringey_gov_uk.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/haringey_gov_uk.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/haringey_gov_uk.py
@@ -38,7 +38,11 @@ def fetch(self):
 
         for service_element in service_elements:
             service_name = service_element.select(".service-name")[0].text.strip()
-            next_service_date = service_element.select("td.next-service")[0]
+
+            next_service_dates = service_element.select("td.next-service")
+            if len(next_service_dates) == 0:
+                continue
+            next_service_date = next_service_dates[0]
 
             next_service_date.span.extract()
 


Test patch: