Repository: alltheplaces/alltheplaces

Issue:
Checkers (ZA) (shop/supermarket)
### Brand name

Checkers

### Wikidata ID

Q5089126

### Store finder url(s)


https://www.checkers.co.za/medias/Store-checkersZA-0.xml

Sitemap regex: `^http:\/\/www\.checkers\.co\.za\/store\/\d+`


Discussion:
Note many of these have LiquorShop (https://github.com/alltheplaces/alltheplaces/issues/7015) located within.
Bot Protection of some kind.
```
from scrapy.spiders import SitemapSpider

from locations.structured_data_spider import StructuredDataSpider


class CheckersZASpider(SitemapSpider, StructuredDataSpider):
    name = "checkers_za"
    item_attributes = {"brand": "Checkers", "brand_wikidata": "Q5089126"}
    sitemap_urls = ["https://www.checkers.co.za/sitemap/medias/Store-checkersZA-0.xml"]
    sitemap_rules = [(r"store\/\d+$", "parse_sd")]
    wanted_types = ["Store"]

```

```

2024-02-11 22:19:57 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.checkers.co.za/store/6501?lat=-33.891&long=18.561> from <GET http://www.checkers.co.za/store/6501?lat=-33.891&long=18.561>
2024-02-11 22:19:58 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.checkers.co.za/store/6462?lat=-33.4627&long=18.729> from <GET http://www.checkers.co.za/store/6462?lat=-33.4627&long=18.729>
2024-02-11 22:19:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.checkers.co.za/store/6446?lat=-33.9062&long=18.5991> from <GET http://www.checkers.co.za/store/6446?lat=-33.9062&long=18.5991>
2024-02-11 22:20:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.checkers.co.za/store/6331?lat=-34.185&long=22.115> from <GET http://www.checkers.co.za/store/6331?lat=-34.185&long=22.115>
2024-02-11 22:20:02 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.checkers.co.za/store/6399?lat=-33.989&long=18.486> from <GET http://www.checkers.co.za/store/6399?lat=-33.989&long=18.486>
2024-02-11 22:20:03 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://euvalidate.perfdrive.com?ssa=5f0fdda0-48c3-3af6-26ba-94802d277f53&ssb=14736264998&ssc=https%3A%2F%2Fwww.checkers.co.za%2Fstore%2F7133%3Flat%3D-25.674%26long%3D28.174&ssi=96792d9f-bklb-4813-71aa-91aaef709823&ssk=contactus@shieldsquare.com&ssm=76230899996646728108023681145124&ssn=443d4aa644eae82b7842ad1b09c71776e2236c278da3-8cc9-7421-dd82cb&sso=7a42f062-771b7e8241403459c88c507991b223696deb01fe9e58a96e&ssp=31673651141707632320170764779516832&ssq=04789239000322924583990003399881770176232&ssr=MjAuMjI3LjE1Mi40NQ==&sst=Mozilla/5.0%20(X11;%20Linux%20x86_64)%20locations/1.0%20(+https://github.com/alltheplaces/alltheplaces;%20framework%202.11.0)&ssw=> from <GET https://www.checkers.co.za/store/7133?lat=-25.674&long=28.174>
2024-02-11 22:20:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://euvalidate.perfdrive.com/robots.txt> (referer: None)
2024-02-11 22:20:03 [protego] DEBUG: Rule at line 9 without any user agent to enforce it on.
```

Resolving patch:
diff --git a/locations/spiders/checkers_za.py b/locations/spiders/checkers_za.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/checkers_za.py
@@ -0,0 +1,67 @@
+from scrapy import Request, Selector
+
+from locations.categories import Categories, apply_category
+from locations.settings import DEFAULT_PLAYWRIGHT_SETTINGS
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class CheckersZASpider(StructuredDataSpider):
+    name = "checkers_za"
+    start_urls = ["https://www.checkers.co.za/sitemap/medias/Store-checkersZA-0.xml"]
+    sitemap_rules = [(r"s^http:\/\/www\.checkers\.co\.za\/store\/\d+$", "parse_sd")]
+    # AWS WAF bot protection appears to be used but can be bypassed with Playwright.
+    is_playwright_spider = True
+    custom_settings = DEFAULT_PLAYWRIGHT_SETTINGS | {"CONCURRENT_REQUESTS": 1, "ROBOTSTXT_OBEY": False}
+
+    def start_requests(self):
+        for sitemap_url in self.start_urls:
+            yield Request(
+                url=sitemap_url,
+                meta={"playwright": True, "playwright_include_page": True},
+                callback=self.parse_sitemap,
+                errback=self.errback_close_page,
+            )
+
+    async def errback_close_page(self, failure):
+        playwright_page = failure.request.meta["playwright_page"]
+        await playwright_page.close()
+
+    async def parse_sitemap(self, response):
+        playwright_page = response.meta["playwright_page"]
+        sitemap = await playwright_page.content()
+        await playwright_page.close()
+        sitemap_locations = Selector(text=sitemap).xpath("//loc/text()").getall()
+        for sitemap_location in sitemap_locations:
+            yield Request(url=sitemap_location.replace("http://", "https://").split("?", 1)[0], callback=self.parse_sd)
+
+    def pre_process_data(self, ld_data):
+        # One openingHourSpecification mixes hours for a
+        # supermarket (first listed) and hours for any additional
+        # stores-in-a-store (pharmacy, liquor shop, etc). Only the
+        # first set of opening hours should be extracted.
+        if ohspec := ld_data.get("openingHoursSpecification"):
+            newohspec = {}
+            for day_hours in ohspec:
+                if day_hours["dayOfWeek"] in newohspec.keys():
+                    break
+                newohspec[day_hours["dayOfWeek"]] = day_hours
+            ld_data["openingHoursSpecification"] = newohspec.values()
+
+    def post_process_item(self, item, response, ld_data):
+        if "Checkers Hyper " in item["name"]:
+            item["brand"] = "Checkers Hyper"
+            item["brand_wikidata"] = "Q116518886"
+            item["name"] = item["name"].replace("Checkers Hyper ", "")
+        elif "Checkers " in item["name"]:
+            item["brand"] = "Checkers"
+            item["brand_wikidata"] = "Q5089126"
+            item["name"] = item["name"].replace("Checkers ", "")
+        else:
+            self.logger.warning("Unknown brand from store name: {}".format(item["name"]))
+            return
+        apply_category(Categories.SHOP_SUPERMARKET, item)
+        item["ref"] = response.url.split("/")[-1]
+        item["website"] = response.xpath('//link[@rel="canonical"]/@href').get()
+        item.pop("facebook", None)
+        item.pop("twitter", None)
+        yield item


Test patch: