Repository: alltheplaces/alltheplaces

Issue:
Only about half of stores returned by costacoffee_gb spider
https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/costacoffee_gb.py

I think the Costa Coffee GB spider is only returning around half of the Costa stores. (My attempt to match the non-express stores from AllThePlaces with OSM data has found only about half of the ~1800 OSM objects with brand:wikidata=Q608845 are matched. See https://osm.mathmos.net/chains/Q608845/ )

For example, the Diss Mere Street store at IP22 4AD is not in the AllThePlaces output: store https://www.costa.co.uk/locations/store-locator/map?latitude=52.3749&longitude=1.1089399999999898&open=1

From the comment at line 17 of costacoffee_gb.py:
`# TODO: Can't figure out how to return more than 5 miles`
it seems you may already be aware of this, but I couldn't see an issue about it, so I thought it best to create one.


Discussion:
We can do `maxrec=0` instead of `maxrec=600` to remove the result limit, but that doesn't help us in this case. If we can't get more than 5 miles out of this API, we need to do more requests, at least in the more populated areas.

@costa-coffee help us out on this? We need an efficient API for all the stores, and ideally the vending machines.
This spider like many similar (Big Mac anyone?) would benefit from adding a geonames dependency to the project. This has cities by population by territory. These brands in general put their outlets where there are people. ALP sits at about 6,500, I sit at about 8,700 with a less scatter gun geographic query approach. _Horses for courses_ - in many cases the scatter gun is just what you want. In other cases scatter gun with an additional layer of finer grain scatter gun in areas of large population (thanks again geonames).

> @costa-coffee help us out on this? We need an efficient API for all the stores, and ideally the vending machines.

@atmos87 ?

I've added geonames "support" as part of https://githhttps://github.com/alltheplaces/alltheplaces/pull/3831ub.com/alltheplaces/alltheplaces/pull/3831

@rjw62 : if the above makes it into the weekly run this week you will have need McDonald's GB output. I believe the current mcdonals UK output is stunted in a similar manner to Costa here.

Resolving patch:
diff --git a/locations/pipelines.py b/locations/pipelines.py
--- a/locations/pipelines.py
+++ b/locations/pipelines.py
@@ -125,14 +125,15 @@ def process_item(self, item, spider):
             spider.crawler.stats.inc_value("atp/field/image/missing")
 
         lat, lon = item["lat"], item["lon"]
-        if not (self.min_lat < float(lat) < self.max_lat):
-            spider.crawler.stats.inc_value("atp/field/lat/invalid")
-        if not (self.min_lon < float(lon) < self.max_lon):
-            spider.crawler.stats.inc_value("atp/field/lon/invalid")
-        if math.fabs(float(lat)) < 0.01:
-            spider.crawler.stats.inc_value("atp/field/lat/invalid")
-        if math.fabs(float(lon)) < 0.01:
-            spider.crawler.stats.inc_value("atp/field/lon/invalid")
+        if lat and lon:
+            if not (self.min_lat < float(lat) < self.max_lat):
+                spider.crawler.stats.inc_value("atp/field/lat/invalid")
+            if not (self.min_lon < float(lon) < self.max_lon):
+                spider.crawler.stats.inc_value("atp/field/lon/invalid")
+            if math.fabs(float(lat)) < 0.01:
+                spider.crawler.stats.inc_value("atp/field/lat/invalid")
+            if math.fabs(float(lon)) < 0.01:
+                spider.crawler.stats.inc_value("atp/field/lon/invalid")
 
         if phone := item.get("phone"):
             if not isinstance(phone, str):
@@ -175,7 +176,10 @@ def process_item(self, item, spider):
         if opening_hours := item.get("opening_hours"):
             if not isinstance(opening_hours, str):
                 spider.crawler.stats.inc_value("atp/field/opening_hours/wrong_type")
-            elif not self.opening_hours_regex.match(opening_hours):
+            elif (
+                not self.opening_hours_regex.match(opening_hours)
+                and opening_hours != "24/7"
+            ):
                 spider.crawler.stats.inc_value("atp/field/opening_hours/invalid")
         else:
             spider.crawler.stats.inc_value("atp/field/opening_hours/missing")
diff --git a/locations/spiders/costacoffee_gb.py b/locations/spiders/costacoffee_gb.py
--- a/locations/spiders/costacoffee_gb.py
+++ b/locations/spiders/costacoffee_gb.py
@@ -1,47 +1,38 @@
 # -*- coding: utf-8 -*-
-import csv
 import scrapy
 
-from locations.items import GeojsonPointItem
+from locations.dict_parser import DictParser
 from locations.hours import OpeningHours
 
 
+def yes_or_no(condition):
+    return "yes" if condition else "no"
+
+
 class CostaCoffeeGBSpider(scrapy.Spider):
     name = "costacoffee_gb"
     item_attributes = {"brand": "Costa Coffee", "brand_wikidata": "Q608845"}
     allowed_domains = ["www.costa.co.uk"]
-    download_delay = 1
+    start_urls = ["https://www.costa.co.uk/api/locations/stores"]
 
-    def start_requests(self):
-        template = "https://www.costa.co.uk/api/locations/stores?latitude={lat}&longitude={lon}&maxrec=600"
-        # TODO: Can't figure out how to return more than 5 miles
+    def parse(self, response):
+        for store_data in response.json()["stores"]:
+            if store_data.get("storeStatus") != "TRADING":
+                continue
 
-        with open(
-            "./locations/searchable_points/eu_centroids_20km_radius_country.csv"
-        ) as points:
-            reader = csv.DictReader(points)
-            for point in reader:
-                if point["country"] == "UK":
-                    yield scrapy.http.Request(
-                        url=template.format(
-                            lat=point["latitude"], lon=point["longitude"]
-                        ),
-                        callback=self.parse,
-                    )
+            store_data["address"] = store_data["storeAddress"]
 
-    def parse(self, response):
-        jsonresponse = response.json()
-        for store_data in jsonresponse["stores"]:
-            addr_full = ", ".join(
+            item = DictParser.parse(store_data)
+
+            item["ref"] = store_data["storeNo8Digit"]
+            item["name"] = store_data["storeNameExternal"]
+            item["street_address"] = ", ".join(
                 filter(
                     None,
                     (
                         store_data["storeAddress"]["addressLine1"],
                         store_data["storeAddress"]["addressLine2"],
                         store_data["storeAddress"]["addressLine3"],
-                        store_data["storeAddress"]["city"],
-                        store_data["storeAddress"]["postCode"],
-                        "United Kingdom",
                     ),
                 ),
             )
@@ -55,57 +46,38 @@ def parse(self, response):
                         store_data["storeOperatingHours"]["close" + day],
                     )
 
-            properties = {
-                "ref": store_data["storeNo8Digit"],
-                "name": store_data["storeNameExternal"],
-                "addr_full": addr_full.strip(),
-                "street_address": store_data["storeAddress"]["addressLine1"],
-                "city": store_data["storeAddress"]["city"],
-                "postcode": store_data["storeAddress"]["postCode"],
-                "country": store_data["storeAddress"]["country"],
-                "lat": float(store_data["latitude"]),
-                "lon": float(store_data["longitude"]),
-                "phone": store_data["telephone"],
-                "opening_hours": opening_hours.as_opening_hours(),
-                "extras": {
-                    "email": store_data["email"],
-                    "store_type": store_data["storeType"],
-                    "storeBusinessModel": store_data["storeBusinessModel"],
-                },
-            }
+            item["opening_hours"] = opening_hours.as_opening_hours()
+            item["extras"] = {"email": store_data["email"]}
 
             for storeFacility in store_data["storeFacilities"]:
                 if storeFacility["name"] == "Wifi":
                     if storeFacility["active"]:
-                        properties["extras"]["internet_access"] = "wlan"
+                        item["extras"]["internet_access"] = "wlan"
                     else:
-                        properties["extras"]["internet_access"] = "no"
+                        item["extras"]["internet_access"] = "no"
                 elif storeFacility["name"] == "Disabled WC":
                     if storeFacility["active"]:
-                        properties["extras"]["toilets"] = "yes"
-                        properties["extras"]["toilets:wheelchair"] = "yes"
+                        item["extras"]["toilets"] = "yes"
+                        item["extras"]["toilets:wheelchair"] = "yes"
                     else:
-                        properties["extras"]["toilets:wheelchair"] = "no"
+                        item["extras"]["toilets:wheelchair"] = "no"
                 elif storeFacility["name"] == "Baby Changing":
-                    if storeFacility["active"]:
-                        properties["extras"]["changing_table"] = "yes"
-                    else:
-                        properties["extras"]["changing_table"] = "no"
+                    item["extras"]["changing_table"] = yes_or_no(
+                        storeFacility["active"]
+                    )
                 elif storeFacility["name"] == "Disabled Access":
-                    if storeFacility["active"]:
-                        properties["extras"]["wheelchair"] = "yes"
-                    else:
-                        properties["extras"]["wheelchair"] = "no"
+                    item["extras"]["wheelchair"] = yes_or_no(storeFacility["active"])
                 elif storeFacility["name"] == "Drive Thru":
-                    if storeFacility["active"]:
-                        properties["extras"]["drive_through"] = "yes"
-                    else:
-                        properties["extras"]["drive_through"] = "no"
+                    item["extras"]["drive_through"] = yes_or_no(storeFacility["active"])
                 elif storeFacility["name"] == "Delivery":
-                    if storeFacility["active"]:
-                        properties["extras"]["delivery"] = "yes"
-                    else:
-                        properties["extras"]["delivery"] = "no"
+                    item["extras"]["delivery"] = yes_or_no(storeFacility["active"])
+
+            if store_data["storeType"] == "COSTA EXPRESS":
+                item["brand"] = "Costa Express"
+                item["extras"]["amenity"] = "vending_machine"
+                item["extras"]["vending"] = "coffee"
+            else:
+                item["extras"]["amenity"] = "cafe"
+                item["extras"]["cuisine"] = "coffee_shop"
 
-            if store_data["storeStatus"] == "TRADING":
-                yield GeojsonPointItem(**properties)
+            yield item


Test patch: