Repository: mampfes/hacs_waste_collection_schedule

Issue:
[Bug]: Ashford [UK] - Page layout changed, parser fails
### I Have A Problem With:

A specific source

### What's Your Problem

Council has updated their page layout slightly after contracting a new waste collector.

The `table` the parser is looking for is no longer present. All data is still available on the page.

### Source (if relevant)

ashford_gov_uk

### Logs

```Shell
failed for source Ashford Borough Council: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/ashford_gov_uk.py", line 85, in fetch raise Exception( Exception: could not get valid data from ashford.gov.uk. is your UPRN (<UPRN REMOVED>) correct for postcode (<POSTCODE REMOVED>)? check https://uprn.uk/<UPRN REMOVED> and https://secure.ashford.gov.uk/wastecollections/collectiondaylookup/
```


### Relevant Configuration

```YAML
# Waste Collection
waste_collection_schedule:
    sources: 
    - name: ashford_gov_uk
      args: 
        uprn: <UPRN REMOVED>
        postcode: <POSTCODE REMOVED>
      customize:
        - type: Recycling (green bin / clear sacks)
          alias: Recycling
        - type: Food waste (black container with orange lid)
          alias: Food Waste
        
sensor:
    - platform: waste_collection_schedule
      name: ashford_gov_uk
```


### Checklist Source Error

- [X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
- [X] Checked that the website of your service provider is still working
- [X] Tested my attributes on the service provider website (if possible)
- [X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

### Checklist Sensor Error

- [X] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

### Required

- [X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
- [X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.


Discussion:
NB: I would create a PR myself, but my test environment is having some issues between Python 3.11 and OpenSSL 3.2.1 on MacOS, not allowing me to test the source.
I have some weird SLL issues with multiple British providers as well. Need to look into a fix/workaround.
I can have a look into it properly tomorrow. I’ve been playing around with it tonight as just a python script by itself as I would like to integrate into another project
From a quick look this morning, it seems as if the tables are still there (one for each bin type).

Just before the error is thrown, if you get it to print out the HTML contents of `soup`, that gives out the page with the select address dropdown still present and no bin data - it looks like the form isn't being filled in correctly. 
Did anyone work out a fix for this issue?
Any work arounds for this? I have time on the weekend to have a look too. 
I had a look but couldn’t work out what was going wrong - the online form is a weird way to get bin data

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/ashford_gov_uk.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/ashford_gov_uk.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/ashford_gov_uk.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/ashford_gov_uk.py
@@ -1,25 +1,17 @@
-import requests
-from waste_collection_schedule import Collection  # type: ignore[attr-defined]
-from bs4 import BeautifulSoup, Tag
 import re
 from datetime import datetime
 
+import requests
+from bs4 import BeautifulSoup
+from waste_collection_schedule import Collection  # type: ignore[attr-defined]
+
 TITLE = "Ashford Borough Council"
 DESCRIPTION = "Source for Ashford Borough Council."
 URL = "https://ashford.gov.uk"
 TEST_CASES = {
-    "100060796052": {
-        "uprn": 100060796052,
-        "postcode": "TN23 3DY"
-    },
-    "100060780440": {
-        "uprn": "100060780440",
-        "postcode": "TN24 9JD"
-    },
-    "100062558476": {
-        "uprn": "100062558476",
-        "postcode": "TN233LX"
-    },
+    "100060796052": {"uprn": 100060796052, "postcode": "TN23 3DY"},
+    "100060780440": {"uprn": "100060780440", "postcode": "TN24 9JD"},
+    "100062558476": {"uprn": "100062558476", "postcode": "TN233LX"},
 }
 
 
@@ -31,7 +23,7 @@
 }
 
 
-API_URL = "https://secure.ashford.gov.uk/wastecollections/collectiondaylookup/"
+API_URL = "https://secure.ashford.gov.uk/waste/collectiondaylookup/"
 
 
 class Source:
@@ -39,56 +31,71 @@ def __init__(self, postcode: str, uprn: str | int):
         self._uprn = str(uprn).strip()
         self._postcode = str(postcode).strip()
 
-        self._post_code_args = {"CollectionDayLookup2$Button_PostCodeSearch": "Continue+>"}
-        self._uprn_args = {"CollectionDayLookup2$Button_SelectAddress": "Continue+>"}
-        self._post_code_args["CollectionDayLookup2$TextBox_PostCode"] = self._postcode
-        self._uprn_args["CollectionDayLookup2$DropDownList_Addresses"] = self._uprn
-
     def fetch(self):
-        # get json file
         s = requests.Session()
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+            "Application": "application/x-www-form-urlencoded",
+        }
+        s.headers.update(headers)
         r = s.get(API_URL)
         r.raise_for_status()
-
         soup: BeautifulSoup = BeautifulSoup(r.text, "html.parser")
 
-        viewstate = soup.find("input", id="__VIEWSTATE")
-        viewstate_generator = soup.find("input", id="__VIEWSTATEGENERATOR")
-        
-        if not viewstate or type(viewstate) != Tag or not viewstate_generator or type(viewstate_generator) != Tag:
-            raise Exception("could not get valid data from ashford.gov.uk")
-
-        self._post_code_args["__VIEWSTATE"] = str(viewstate["value"])
-
-        self._post_code_args["__VIEWSTATEGENERATOR"] = str(viewstate_generator["value"])
-        self._uprn_args["__VIEWSTATEGENERATOR"] = str(viewstate_generator["value"])
-
-        r = s.post(API_URL, data=self._post_code_args)
+        args = {}
+        for input_tag in soup.find_all("input"):
+            if not input_tag.get("name"):
+                continue
+            args[input_tag["name"]] = input_tag.get("value")
+        args["ctl00$ContentPlaceHolder1$CollectionDayLookup2$HiddenField_UPRN"] = ""
+        args[
+            "ctl00$ContentPlaceHolder1$CollectionDayLookup2$TextBox_PostCode"
+        ] = self._postcode
+        args[
+            "ctl00$ContentPlaceHolder1$CollectionDayLookup2$Button_PostCodeSearch"
+        ] = "Continue+>"
+        args["__EVENTTARGET"] = ""
+        args["__EVENTARGUMENT"] = ""
+
+        r = s.post(API_URL, data=args)
         r.raise_for_status()
 
         soup: BeautifulSoup = BeautifulSoup(r.text, "html.parser")
-        viewstate = soup.find("input", id="__VIEWSTATE")
-        if not viewstate or type(viewstate) != Tag:
-            raise Exception("could not get valid data from ashford.gov.uk")
-
-        self._uprn_args["__VIEWSTATE"] = str(viewstate["value"])
-
-        r = s.post(API_URL, data=self._uprn_args)
+        args = {}
+        for input_tag in soup.find_all("input"):
+            if not input_tag.get("name"):
+                continue
+            args[input_tag["name"]] = input_tag.get("value")
+        args[
+            "ctl00$ContentPlaceHolder1$CollectionDayLookup2$DropDownList_Addresses"
+        ] = self._uprn
+
+        args[
+            "ctl00$ContentPlaceHolder1$CollectionDayLookup2$Button_PostCodeSearch"
+        ] = "Continue+>"
+        del args["ctl00$ContentPlaceHolder1$CollectionDayLookup2$Button_SelectAddress"]
+        del args["ctl00$ContentPlaceHolder1$CollectionDayLookup2$Button_PostCodeSearch"]
+        args[
+            "ctl00$ContentPlaceHolder1$CollectionDayLookup2$Button_SelectAddress"
+        ] = "Continue+>"
+
+        r = s.post(API_URL, data=args)
         if r.status_code != 200:
             raise Exception(
-                f"could not get correct data for your postcode ({self._postcode}). check {API_URL} to validate your arguments.")
+                f"could not get correct data for your postcode ({self._postcode}). check {API_URL} to validate your arguments."
+            )
 
         soup: BeautifulSoup = BeautifulSoup(r.text, "html.parser")
-
         bin_tables = soup.find_all("table")
         if bin_tables == []:
             raise Exception(
-                f"could not get valid data from ashford.gov.uk. is your UPRN ({self._uprn}) correct for postcode ({self._postcode})? check https://uprn.uk/{self._uprn} and {API_URL}")
+                f"could not get valid data from ashford.gov.uk. is your UPRN ({self._uprn}) correct for postcode ({self._postcode})? check https://uprn.uk/{self._uprn} and {API_URL}"
+            )
 
         entries = []
         for bin_table in bin_tables:
-            bin_text = bin_table.find(
-                "td", id=re.compile("CollectionDayLookup2_td_"))
+            bin_text = bin_table.find("td", id=re.compile("CollectionDayLookup2_td_"))
             if not bin_text:
                 continue
 
@@ -98,13 +105,14 @@ def fetch(self):
                 continue
             bin_type: str = bin_type_soup.text.strip()
 
-            date_soup = bin_text.find("span", id=re.compile(r"CollectionDayLookup2_Label_\w*_Date"))
-            if not date_soup or not " " in date_soup.text.strip():
+            date_soup = bin_text.find(
+                "span", id=re.compile(r"CollectionDayLookup2_Label_\w*_Date")
+            )
+            if not date_soup or " " not in date_soup.text.strip():
                 continue
             date_str: str = date_soup.text.strip()
             try:
-                date = datetime.strptime(
-                    date_str.split(" ")[1], "%d/%m/%y").date()
+                date = datetime.strptime(date_str.split(" ")[1], "%d/%m/%Y").date()
 
             except ValueError:
                 continue


Test patch: