Repository: alltheplaces/alltheplaces

Issue:
Fix Costa Coffee (US) which has switched from Amasty to a JSON API
In the latest weekly run, Costa Coffee (costacoffee_us) is now broken with the Amasty "amlocator" AJAX endpoint no longer existing. In it's place appears to be a JSON API that returns store details (locations and addresses):

https://us.costacoffee.com/api/cf/?locale=en-US&include=2&content_type=storeLocatorStore&limit=500&fields.location[near]=33.77804102,-84.38068933

This appears to be a fairly easy fix to rewrite the spider to use the new JSON API (with a higher limit than 500).


Discussion:
And pages [nicely](https://us.costacoffee.com/api/cf/?locale=en-US&include=2&content_type=storeLocatorStore&limit=500&skip=500).

I've seen this `fields`/`sys`/`metadata` thing before, but I don't think I looked into the details. We should keep an out in case it's common and deserves shared code.
locations/spiders/costacoffee_ie.py:20:                "ref": store_data["sys"]["id"],
locations/spiders/tele2_se.py:34:                    "ref": store.get("sys").get("id"),
locations/spiders/brewdog.py:21:            item["ref"] = bar["sys"]["id"]

tele2_se doesn't appear to be the same

costacoffee_ie looks the same.

brewdog looks the same except it is provided as a JSON blob in the store locator HTML page (React __NEXT_DATA__ used).

Resolving patch:
diff --git a/locations/spiders/costacoffee_us.py b/locations/spiders/costacoffee_us.py
--- a/locations/spiders/costacoffee_us.py
+++ b/locations/spiders/costacoffee_us.py
@@ -1,49 +1,38 @@
-import json
-import re
-
-import scrapy
-from scrapy import Selector
+from scrapy import Spider
+from scrapy.http import JsonRequest
 
 from locations.categories import Categories, apply_category
-from locations.items import Feature
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_FULL, OpeningHours
 
 
-class CostaCoffeeUSSpider(scrapy.Spider):
+class CostaCoffeeUSSpider(Spider):
     name = "costacoffee_us"
     item_attributes = {"brand": "Costa Coffee", "brand_wikidata": "Q608845"}
     allowed_domains = ["us.costacoffee.com"]
-    start_urls = ["https://us.costacoffee.com/amlocator/index/ajax"]
-
-    def parse(self, response):
-        script = response.xpath('//script[contains(text(), "amLocator")]/text()').extract_first()
-
-        start = script.index("jsonLocations: ") + len("jsonLocations: ")
-        stop = script.index("imageLocations")
-
-        locations = script[start:stop].strip().strip(",")
-        items = json.loads(locations)["items"]
-
-        for store in items:
-            item = Feature()
-            item["ref"] = store["id"]
-            item["lat"] = store["lat"]
-            item["lon"] = store["lng"]
+    start_urls = ["https://us.costacoffee.com/api/cf/?content_type=storeLocatorStore"]
+    page_size = 1000
 
-            html = Selector(text=store["popup_html"])
-
-            item["name"] = html.xpath('//*[@class="amlocator-title"]/text()').get()
-
-            for line in html.xpath('//div[@class="amlocator-info-popup"]/text()').getall():
-                line = line.strip()
-                if m := re.match(r"City: (.*)", line):
-                    item["city"] = m.group(1)
-                elif m := re.match(r"Zip: (.*)", line):
-                    item["postcode"] = m.group(1)
-                elif m := re.match(r"Address: (.*)", line):
-                    item["street_address"] = m.group(1)
-                elif m := re.match(r"State: (.*)", line):
-                    item["state"] = m.group(1)
+    def start_requests(self):
+        for url in self.start_urls:
+            yield JsonRequest(url=f"{url}&limit={self.page_size}")
 
+    def parse(self, response):
+        for location in response.json()["items"]:
+            item = DictParser.parse(location["fields"])
+            item["ref"] = location["sys"]["id"]
+            item["addr_full"] = location["fields"]["storeAddress"]
+            item["opening_hours"] = OpeningHours()
+            for day_name in [s.lower() for s in DAYS_FULL]:
+                open_time = location["fields"].get(f"{day_name}Opening")
+                close_time = location["fields"].get(f"{day_name}Closing")
+                if open_time and "24 HOURS" in open_time.upper():
+                    item["opening_hours"].add_range(day_name, "00:00", "24:00")
+                elif open_time and close_time:
+                    item["opening_hours"].add_range(day_name, open_time, close_time)
             apply_category(Categories.COFFEE_SHOP, item)
-
             yield item
+
+        offset = response.json()["skip"]
+        if offset + response.json()["limit"] < response.json()["total"]:
+            yield JsonRequest(url=f"{response.request.url}&limit={self.page_size}&offset={offset}")


Test patch: