Repository: mampfes/hacs_waste_collection_schedule

Issue:
Request for new provider, West Devon UK
Please add the source for West Devon
https://westdevon.fccenvironment.co.uk/mycollections

Thanks,
Tim


Discussion:
fccenvironment.co.uk is the same service provider Harborough District Council uses, but the West Devon implementation looks to be doing things a different way.  I get 403 errors and _'No direct script access allowed'_ and _'The action you have requested is not allowed'_ messages when trying to adapt the existing fccenvironment.co.uk approach.
Ok thanks for looking, I guess I wait for a different approach?
@mampfes it looks like a ton of local authorities use `fccenvironment.co.uk` in one way or another, however, the [existing module](https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/custom_components/waste_collection_schedule/waste_collection_schedule/source/fccenvironment_co_uk.py) is written in a way very specific to Harborough. Changing this module might break existing users depending on how we move forward with this, I see two choices:

1) Rename `fccenvironment_co_uk` to `fccenvironment_co_uk_harborough`, create a new source called `fccenvironment_co_uk_westdevon`, symlink the renamed files. This _should_ mean existing users continue to function as is with no config changes.
2) Add multiple methods of scraping to the existing `fccenvironment_co_uk` module, this requires a config change for end users as they'll need to provide a `district` or similar config option to let the source know _what_ to scrape.

Personally, 1) feels like the correct method here. Let me know and I'll open a PR.

I guess it depends on how many of the ~60 councils ultimately end up being supported - a breaking change introducing the need for a `district` argument may make sense for long-term maintenance if there's a desire to keep the number of .py and .md files to a minimum.

A number of the existing sources have multiple councils/municipalities per file, [a_region_ch](https://github.com/mampfes/hacs_waste_collection_schedule/blob/master/custom_components/waste_collection_schedule/waste_collection_schedule/source/a_region_ch.py), so there's already a precedence for taking that approach.
Alternatively, introduce a `district`, but have the default value as "harborough". That'd also allow the current users to continue with the existing configs.

From what I could see when I originally had a quick look at West Devon, both that and Harborough just need `uprn`'s supplied, but the endpoints were different:
 - https://www.fccenvironment.co.uk/harborough/detail-address
 - https://westdevon.fccenvironment.co.uk/ajaxprocessor/getcollectiondetails


I completely agree with @dt215git.

I would prefer to create one source if multiple councils share the same service. The main problem with this approach is that the user has to know the name of the service - typically they start searching with the name of their council/municipality/etc. What we did in the recent past was to add both to the documentation: 
- One markdown file which describes the source and all supported councils/...
- A link in README.md to the source markdown (with the name of the source).
- Extra links for supported councils in README.md which just link to the source markdown.
As this is an annoying activity, I dream of a script or s.th. similar which creates the documentation links automatically...

If you have any ideas please let me know - but let us discuss in a separate thread.
@mothy-tim - could you give some advice here please? I've finished the scraper but the page returns the same collection multiple times:

![Screenshot 2022-12-27 at 14 07 21](https://user-images.githubusercontent.com/6557665/209678201-859d86bc-4abd-4d27-87ff-9d176e0dcaf6.png)

I'm just trying to figure out if I should treat these three "Recycling and Food Collection Service" entries as duplicates or if they're different poorly labeled services. If they are different, can you advise what each service actually is?

Resolving patch:
diff --git a/custom_components/waste_collection_schedule/waste_collection_schedule/source/fccenvironment_co_uk.py b/custom_components/waste_collection_schedule/waste_collection_schedule/source/fccenvironment_co_uk.py
--- a/custom_components/waste_collection_schedule/waste_collection_schedule/source/fccenvironment_co_uk.py
+++ b/custom_components/waste_collection_schedule/waste_collection_schedule/source/fccenvironment_co_uk.py
@@ -1,74 +1,127 @@
-import logging
-import requests
+from urllib.parse import urlparse
 
+import requests
 from bs4 import BeautifulSoup
-from datetime import datetime
+from dateutil import parser
 from waste_collection_schedule import Collection
 
 TITLE = "fccenvironment.co.uk"
 
-DESCRIPTION = (
-    """Consolidated source for waste collection services for ~60 local authorities.
-        Currently supports:
-        Market Harborough
-        """
-)
+DESCRIPTION = """
+    Consolidated source for waste collection services for ~60 local authorities.
+    Currently supports:
+    West Devon (Generic Provider)
+    South Hams (Generic Provider)
+    Market Harborough (Custom Provider)
+    """
 
 URL = "https://fccenvironment.co.uk"
 
 TEST_CASES = {
-    "Test_001" : {"uprn": "100030491624"},
-    "Test_002": {"uprn": "100030491614"},
-    "Test_003": {"uprn": "100030493289"},
-    "Test_004": {"uprn": "200001136341"}
+    "14_LE16_9QX": {"uprn": "100030491624"},  # region ommited to test default values
+    "4_LE16_9QX": {"uprn": "100030491614", "region": "harborough"},
+    "16_LE16_7NA": {"uprn": "100030493289", "region": "harborough"},
+    "10_LE16_8ER": {"uprn": "200001136341", "region": "harborough"},
+    "9_PL20_7SH": {"uprn": "10001326315", "region": "westdevon"},
+    "3_PL20_7RY": {"uprn": "10001326041", "region": "westdevon"},
+    "2_PL21_9BN": {"uprn": "100040279446", "region": "southhams"},
+    "4_SL21_0HZ": {"uprn": "100040281987", "region": "southhams"},
 }
 
-
 ICONS = {
-    "NON-RECYCLABLE WASTE BIN COLLECTION": "mdi:trash-can",
-    "RECYCLING COLLECTION": "mdi:recycle",
-    "GARDEN WASTE COLLECTION": "mdi:leaf",
+    "Refuse": "mdi:trash-can",
+    "Recycling": "mdi:recycle",
+    "Garden": "mdi:leaf",
 }
 
 
-_LOGGER = logging.getLogger(__name__)
-
-
 class Source:
-    def __init__(self, uprn=None):
-        self._uprn = uprn
-
-    def fetch(self):
-
-        s = requests.Session()
+    def __init__(self, uprn: str, region: str = "harborough") -> None:
+        self.uprn = uprn
+        self.region = region
+
+    def getcollectiondetails(self, endpoint: str) -> list[Collection]:
+        domain = urlparse(endpoint).netloc
+        session = requests.Session()
+        cookies = session.get(f"https://{domain}/")
+        response = session.post(
+            endpoint,
+            headers={
+                "x-requested-with": "XMLHttpRequest",
+            },
+            data={
+                "fcc_session_token": cookies.cookies["fcc_session_cookie"],
+                "uprn": self.uprn,
+            },
+        )
+        results = {}
+        for item in response.json()["binCollections"]["tile"]:
+            try:
+                soup = BeautifulSoup(item[0], "html.parser")
+                date = parser.parse(soup.find_all("b")[2].text.split(",")[1].strip()).date()
+                service = soup.text.split("\n")[0]
+            except parser._parser.ParserError:
+                continue
 
-        if self._uprn:
-            # POST request returns schedule for matching uprn
-            payload = {
-                "Uprn": self._uprn
-            }
-            r = s.post("https://www.fccenvironment.co.uk/harborough/detail-address", data = payload)
-            responseContent = r.text
+            """
+            Handle duplication before creating the list of Collections
+            """
+            for type in ICONS:
+                if type in service:
+                    if type in results.keys():
+                        if date < results[type]:
+                            results[type] = date
+                    else:
+                        results[type] = date
 
         entries = []
-        # Extract waste types and dates from responseContent
-        soup = BeautifulSoup(responseContent, "html.parser")
-        services = soup.find("div", attrs={"class": "blocks block-your-next-scheduled-bin-collection-days"})
-        items = services.find_all("li")
-        for item in items:
-            date_text = item.find("span", attrs={"class": "pull-right"}).text.strip()
-            try:
-                date = datetime.strptime(date_text, "%d %B %Y").date()
-            except ValueError:
-                continue
-            else:
-                waste_type = item.text.split(' (')[0]
-                entries.append(
-                    Collection(
-                        date=date,
-                        t=waste_type,
-                        icon=ICONS.get(waste_type.upper()),    
-                    )
+        for result in results:
+            entries.append(
+                Collection(
+                    date=results[result],
+                    t=result,
+                    icon=ICONS[result],
                 )
+            )
+        return entries
 
+    def harborough(self) -> list[Collection]:
+        _icons = {
+            "NON-RECYCLABLE WASTE BIN COLLECTION": "mdi:trash-can",
+            "RECYCLING COLLECTION": "mdi:recycle",
+            "GARDEN WASTE COLLECTION": "mdi:leaf",
+        }  # Custom icons to avoid a breaking change
+        r = requests.post("https://www.fccenvironment.co.uk/harborough/detail-address", data={"Uprn": self.uprn})
+        soup = BeautifulSoup(r.text, "html.parser")
+        services = soup.find("div", attrs={"class": "blocks block-your-next-scheduled-bin-collection-days"}).find_all(
+            "li"
+        )
+        entries = []
+        for service in services:
+            for type in _icons:
+                if type.lower() in service.text.lower():
+                    try:
+                        date = parser.parse(service.find("span", attrs={"class": "pull-right"}).text.strip()).date()
+                    except parser._parser.ParserError:
+                        continue
+
+                    entries.append(
+                        Collection(
+                            date=date,
+                            t=type,
+                            icon=_icons[type.upper()],
+                        )
+                    )
         return entries
+
+    def fetch(self) -> list[Collection]:
+        if self.region == "harborough":
+            return self.harborough()
+        elif self.region == "westdevon":
+            return self.getcollectiondetails(
+                endpoint="https://westdevon.fccenvironment.co.uk/ajaxprocessor/getcollectiondetails"
+            )
+        elif self.region == "southhams":
+            return self.getcollectiondetails(
+                endpoint="https://waste.southhams.gov.uk/mycollections/getcollectiondetails"
+            )


Test patch: