Repository: mampfes/hacs_waste_collection_schedule

Issue:
Cannot fetch from West Berkshire Council
I have never been able to fetch using the West Berkshire Council integration and see the following in the logs:
fetch failed for source West Berkshire Council: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/westberks_gov_uk.py", line 81, in fetch if not rubbish_data['result']['nextRubbishDateSubText']: KeyError: 'nextRubbishDateSubText'


Discussion:
Running test_source.py manually showed an error "No module named 'requests" - I manually ran pip install requests and now test_source.py seems to be returning results (for my URPN that I have edited into the source file manually), but I still see the same error in the logs for the actual integration?
Please share your configuration. The defined test cases in the source file are working, therefore it probably has something to do with your location.

```
./test_sources.py -s westberks_gov_uk -l
Testing source westberks_gov_uk ...
  found 2 entries for known_uprn
    2023-03-31 : Rubbish
    2023-03-24 : Recycling
  found 2 entries for unknown_uprn_by_name
    2023-03-31 : Rubbish
    2023-03-24 : Recycling
  found 2 entries for unknown_uprn_by_number
    2023-04-05 : Rubbish
    2023-03-29 : Recycling
  found 2 entries for unknown_uprn_business
    2023-04-05 : Rubbish
    2023-03-29 : Recycling
```
I'm using
waste_collection_schedule:
  sources:
    - name: westberks_gov_uk
      args:
        uprn: 100081026602

Manually editing the uprn into westberks_gov_uk as a test case still works in test_sources.py, but it doesn't work when trying to use it properly through configuration.yaml . I'm not sure how I can debug it more, there most be something going wrong in my environment...
Seemed to work okay for me

```yaml
waste_collection_schedule:
  sources:
    - name: westberks_gov_uk
      args:
        uprn: "100081026602"

sensor:
  - platform: waste_collection_schedule
    name: WestBerks_Waste
    details_format: appointment_types
    add_days_to: true
```

![image](https://user-images.githubusercontent.com/86194065/227596715-0fc532ee-9dc4-473a-862b-99391fb64bf6.png)

Unfortunately no dice. I’m getting the original error with the exact same config. Is there any other debugging that can be turned on, or the same fetch run manually from the shell somehow?
What type of HA [installation](https://www.home-assistant.io/installation/) are you running?
A supervised Home Assistant deployed from the VM image.
@Quark999 Did you tried it with quotation marks? (as in dt215git's example)?

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/westberks_gov_uk.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/westberks_gov_uk.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/westberks_gov_uk.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/westberks_gov_uk.py
@@ -1,9 +1,8 @@
+import json
+import time
 from datetime import date, datetime
 
-import time
-import json
 import requests
-
 from waste_collection_schedule import Collection
 
 TITLE = "West Berkshire Council"
@@ -11,9 +10,13 @@
 URL = "https://westberks.gov.uk"
 TEST_CASES = {
     "known_uprn": {"uprn": "100080241094"},
-    "unknown_uprn_by_name": {"postcode": "RG7 6NZ", "housenumberorname": "PARROG HOUSE"},
+    "known_uprn as number": {"uprn": 100081026602},
+    "unknown_uprn_by_name": {
+        "postcode": "RG7 6NZ",
+        "housenumberorname": "PARROG HOUSE",
+    },
     "unknown_uprn_by_number": {"postcode": "RG18 4QU", "housenumberorname": "6"},
-    "unknown_uprn_business": {"postcode": "RG18 4GE", "housenumberorname": "3"}
+    "unknown_uprn_business": {"postcode": "RG18 4GE", "housenumberorname": "3"},
 }
 
 ICON_MAP = {
@@ -28,11 +31,12 @@
 
 COLLECTIONS = {"Rubbish", "Recycling"}
 
+
 class Source:
     def __init__(
         self, uprn=None, postcode=None, housenumberorname=None
     ):  # argX correspond to the args dict in the source configuration
-        self._uprn = uprn
+        self._uprn = str(uprn).zfill(12) if uprn is not None else None
         self._postcode = postcode
         self._housenumberorname = housenumberorname
 
@@ -43,22 +47,28 @@ def fetch(self):
         # Find the UPRN based on the postcode and the property name/number
         if self._uprn is None:
             self._postcode = self._postcode.strip()
-            jsonrpc = {"id": str(int(time.time())),"method": "location.westberks.echoPostcodeFinder","params": {"provider": "",  "postcode": self._postcode}}
-            args = { "callback": "HomeAssistant", "jsonrpc": json.dumps(jsonrpc), "_": 0 }
-            r = requests.get(SEARCH_URLS["uprn_search"], params = args)
-            
+            jsonrpc = {
+                "id": str(int(time.time())),
+                "method": "location.westberks.echoPostcodeFinder",
+                "params": {"provider": "", "postcode": self._postcode},
+            }
+            args = {"callback": "HomeAssistant", "jsonrpc": json.dumps(jsonrpc), "_": 0}
+            r = requests.get(SEARCH_URLS["uprn_search"], params=args)
+
             # I don't really know python so there *must* be a better way to do these four lines!
             response_str = r.content.decode("utf-8")
-            data_length = len(response_str) -1
+            data_length = len(response_str) - 1
             response_sub = response_str[14:data_length]
             address_data = json.loads(response_sub)
-            
-            propertyUprns = address_data['result']
+
+            propertyUprns = address_data["result"]
             for match in propertyUprns:
-                if match['line1'].startswith(self._housenumberorname):
+                if match["line1"].startswith(self._housenumberorname):
+                    self._uprn = match["udprn"]
+                if match["buildingnumber"].startswith(
+                    self._housenumberorname
+                ):  # no evidence (yet) that their database uses this
                     self._uprn = match["udprn"]
-                if match['buildingnumber'].startswith(self._housenumberorname): # no evidence (yet) that their database uses this
-                    self._uprn = match["udprn"]                    
 
         entries = []
 
@@ -66,11 +76,21 @@ def fetch(self):
         if self._uprn is not None:
 
             # POST request - one for each type as it is a separate method in the api
-            rubbish_args = {"jsonrpc":"2.0","id":str(int(time.time())) ,"method":"goss.echo.westberks.forms.getNextRubbishCollectionDate","params":{"uprn":self._uprn}}
-            recycling_args = {"jsonrpc":"2.0","id":str(int(time.time())),"method":"goss.echo.westberks.forms.getNextRecyclingCollectionDate","params":{"uprn":self._uprn}}
-            r = session.post(SEARCH_URLS["collection_search"], json = rubbish_args)
+            rubbish_args = {
+                "jsonrpc": "2.0",
+                "id": str(int(time.time())),
+                "method": "goss.echo.westberks.forms.getNextRubbishCollectionDate",
+                "params": {"uprn": self._uprn},
+            }
+            recycling_args = {
+                "jsonrpc": "2.0",
+                "id": str(int(time.time())),
+                "method": "goss.echo.westberks.forms.getNextRecyclingCollectionDate",
+                "params": {"uprn": self._uprn},
+            }
+            r = session.post(SEARCH_URLS["collection_search"], json=rubbish_args)
             rubbish_data = json.loads(r.content)
-            r = session.post(SEARCH_URLS["collection_search"], json = recycling_args)
+            r = session.post(SEARCH_URLS["collection_search"], json=recycling_args)
             recycling_data = json.loads(r.content)
 
             # if subtext is empty, use datetext
@@ -78,35 +98,51 @@ def fetch(self):
 
             # Extract dates from json
             waste_type = "Rubbish"
-            if not rubbish_data['result']['nextRubbishDateSubText']:
-                dt_str = rubbish_data['result']['nextRubbishDateText'] + " " + str(date.today().year)
+            if not rubbish_data["result"]["nextRubbishDateSubText"]:
+                dt_str = (
+                    rubbish_data["result"]["nextRubbishDateText"]
+                    + " "
+                    + str(date.today().year)
+                )
             else:
-                if len(rubbish_data['result']['nextRubbishDateText'])<12:
-                    dt_str = rubbish_data['result']['nextRubbishDateSubText'] + " " + str(date.today().year)
-            dt_zulu = datetime.strptime(dt_str, '%A %d %B %Y') 
+                if len(rubbish_data["result"]["nextRubbishDateText"]) < 12:
+                    dt_str = (
+                        rubbish_data["result"]["nextRubbishDateSubText"]
+                        + " "
+                        + str(date.today().year)
+                    )
+            dt_zulu = datetime.strptime(dt_str, "%A %d %B %Y")
             dt_local = dt_zulu.astimezone(None)
             entries.append(
                 Collection(
                     date=dt_local.date(),
                     t=waste_type,
-                    icon=ICON_MAP.get(waste_type.upper()),    
+                    icon=ICON_MAP.get(waste_type.upper()),
                 )
             )
 
             waste_type = "Recycling"
-            if not recycling_data['result']['nextRecyclingDateSubText']:
-                dt_str = recycling_data['result']['nextRecyclingDateText'] + " " + str(date.today().year)
+            if not recycling_data["result"]["nextRecyclingDateSubText"]:
+                dt_str = (
+                    recycling_data["result"]["nextRecyclingDateText"]
+                    + " "
+                    + str(date.today().year)
+                )
             else:
-                if len(recycling_data['result']['nextRecyclingDateText'])<12:
-                    dt_str = recycling_data['result']['nextRecyclingDateSubText'] + " " + str(date.today().year)
-            dt_zulu = datetime.strptime(dt_str, '%A %d %B %Y')
+                if len(recycling_data["result"]["nextRecyclingDateText"]) < 12:
+                    dt_str = (
+                        recycling_data["result"]["nextRecyclingDateSubText"]
+                        + " "
+                        + str(date.today().year)
+                    )
+            dt_zulu = datetime.strptime(dt_str, "%A %d %B %Y")
             dt_local = dt_zulu.astimezone(None)
             entries.append(
                 Collection(
                     date=dt_local.date(),
                     t=waste_type,
-                    icon=ICON_MAP.get(waste_type.upper()),    
+                    icon=ICON_MAP.get(waste_type.upper()),
                 )
-            )        
-            
+            )
+
         return entries


Test patch: