internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
Domain missing from Holland & Barrett website URLs
In the holland_and_barrett spider results, the website values returned are missing the domain, e.g. `"website": "/stores/aylesbury-3180/"`. This is what's in the code that the scraper is reading. But presumably AllThePlaces should return a fully qualified url, i.e. `https://www.hollandandbarrett.com/stores/aylesbury-3180/` in this case.

I don't know what the micordata etc standards say about whether relative URLs are allowed, but perhaps the framework code could be modified to automatically complete the URL of the page if a relative URL is harvested.


Discussion:

Resolving patch:
diff --git a/locations/spiders/holland_and_barrett.py b/locations/spiders/holland_and_barrett.py
--- a/locations/spiders/holland_and_barrett.py
+++ b/locations/spiders/holland_and_barrett.py
@@ -19,4 +19,6 @@ class HollandAndBarrettSpider(SitemapSpider):
     download_delay = 1.0
 
     def parse(self, response):
-        yield LinkedDataParser.parse(response, "LocalBusiness")
+        item = LinkedDataParser.parse(response, "LocalBusiness")
+        item["website"] = response.urljoin(item["website"])
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Hungry Horse Spider Missing lots of branches
The hungry_horse spider is currently only returning 27 pubs. I think it should be more like 250. It looks like the regex used to filter pages from the side-map is only set to find a subset of the pages.  

The URLs of the pubs take the form `https://www.hungryhorse.co.uk/pubs/<county>/<pub>/`.

The current regex is `r"https://www.hungryhorse.co.uk/pubs/(west-[\w-]+)/([\w-]+)/$"` which thus only finds pubs in counties starting with "west-". 

Presumably the regex should be replaced with `r"https://www.hungryhorse.co.uk/pubs/([\w-]+)/([\w-]+)/$"` to capture all counties.


Discussion:
Yep, thank you. I got 252 with the better regex. I guess I changed to when I was checking the `sitemap_filter`, but never changed it back. woops.

Resolving patch:
diff --git a/locations/spiders/hungry_horse.py b/locations/spiders/hungry_horse.py
deleted file mode 100644
--- a/locations/spiders/hungry_horse.py
+++ /dev/null
@@ -1,47 +0,0 @@
-import json
-
-from scrapy.spiders import SitemapSpider
-
-from locations.items import GeojsonPointItem
-
-
-class HungryHorseSpider(SitemapSpider):
-    name = "hungry_horse"
-    item_attributes = {"brand": "Hungry Horse", "brand_wikidata": "Q5943510"}
-    allowed_domains = ["www.hungryhorse.co.uk"]
-    sitemap_urls = ["https://www.hungryhorse.co.uk/xml-sitemap"]
-    sitemap_rules = [
-        (
-            r"https://www.hungryhorse.co.uk/pubs/(west-[\w-]+)/([\w-]+)/$",
-            "parse",
-        ),
-    ]
-    download_delay = 0.5
-
-    def sitemap_filter(self, entries):
-        for entry in entries:
-            if (
-                entry["loc"]
-                != "https://www.hungryhorse.co.uk/pubs/west-lothian/lime-kiln/"
-            ):
-                yield entry
-
-    def parse(self, response):
-        ld = json.loads(
-            response.xpath('//script[@type="application/ld+json"]/text()').get()
-        )
-        if not ld.get("branchCode"):
-            return
-        properties = {
-            "ref": ld["branchCode"],
-            "website": response.request.url,
-            "name": ld["name"],
-            "phone": ld["telephone"],
-            "opening_hours": "; ".join(ld["openingHours"]),
-            "lat": ld["geo"]["latitude"],
-            "lon": ld["geo"]["longitude"],
-            "addr_full": ld["address"] + ", United Kingdom",
-            "country": "GB",
-        }
-
-        yield GeojsonPointItem(**properties)
diff --git a/locations/spiders/hungry_horse_gb.py b/locations/spiders/hungry_horse_gb.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/hungry_horse_gb.py
@@ -0,0 +1,11 @@
+from scrapy.spiders import SitemapSpider
+
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class HungryHorseGBSpider(SitemapSpider, StructuredDataSpider):
+    name = "hungry_horse_gb"
+    item_attributes = {"brand": "Hungry Horse", "brand_wikidata": "Q5943510"}
+    allowed_domains = ["www.hungryhorse.co.uk"]
+    sitemap_urls = ["https://www.hungryhorse.co.uk/xml-sitemap"]
+    sitemap_rules = [(r"/pubs/([\w-]+)/([\w-]+)/$", "parse_sd")]


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Domains missing from New Look websites
The new_look_gb.py spider is returning websites that are missing the domain name. This is because that's how the website appears in the schema.org block on the pages being scraped. e.g. we have `"url":"/uk/store/Beccles-Beccles-GB-1775"`.

The scheme and domain `https://www.newlook.com` needs to be prepended to each of the returned URLs.

This is the same issue as in #4302 but for a different spider.


Discussion:

Resolving patch:
diff --git a/locations/spiders/new_look_gb.py b/locations/spiders/new_look_gb.py
--- a/locations/spiders/new_look_gb.py
+++ b/locations/spiders/new_look_gb.py
@@ -15,3 +15,7 @@ def sitemap_filter(self, entries):
         for entry in entries:
             if "closed" not in entry["loc"].lower():
                 yield entry
+
+    def inspect_item(self, item, response):
+        item["website"] = response.urljoin(item["website"])
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Homebase spider webpage regex is too restrictive
The homebase_gb_ie.py spider contains a regex in sitemap_rules to restrict things to store pages:
`sitemap_rules = [(r"https:\/\/store\.homebase\.co\.uk\/[-\w]+\/[-.\w]+$", "parse_sd")]`

This regex is slightly too strict, as there's a store with a "." in the place level: https://store.homebase.co.uk/st.-albans/the-courtyard-alban-park , which is currently not returned.

To include this store, the regex should presumably be changed to
`sitemap_rules = [(r"https:\/\/store\.homebase\.co\.uk\/[-.\w]+\/[-.\w]+$", "parse_sd")]`


Discussion:

Resolving patch:
diff --git a/locations/spiders/homebase_gb_ie.py b/locations/spiders/homebase_gb_ie.py
--- a/locations/spiders/homebase_gb_ie.py
+++ b/locations/spiders/homebase_gb_ie.py
@@ -7,5 +7,5 @@ class HomebaseGBIESpider(SitemapSpider, StructuredDataSpider):
     name = "homebase_gb_ie"
     item_attributes = {"brand": "Homebase", "brand_wikidata": "Q9293447"}
     sitemap_urls = ["https://store.homebase.co.uk/robots.txt"]
-    sitemap_rules = [(r"https:\/\/store\.homebase\.co\.uk\/[-\w]+\/[-.\w]+$", "parse_sd")]
+    sitemap_rules = [(r"https:\/\/store\.homebase\.co\.uk\/[-.\w]+\/[-.\w]+$", "parse_sd")]
     skip_auto_cc = True


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Guardian (Australian pharmacy/chemist)
### Brand name

Guardian

### Wikidata ID

Q63367814

### Store finder url(s)

https://www.guardianpharmacies.com.au/store-locator/vic
https://www.guardianpharmacies.com.au/store-locator/nsw
https://www.guardianpharmacies.com.au/store-locator/qld
https://www.guardianpharmacies.com.au/store-locator/tas
https://www.guardianpharmacies.com.au/store-locator/act
https://www.guardianpharmacies.com.au/store-locator/wa
https://www.guardianpharmacies.com.au/store-locator/sa
https://www.guardianpharmacies.com.au/store-locator/nt


Discussion:

Resolving patch:
diff --git a/locations/spiders/guardian_au.py b/locations/spiders/guardian_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/guardian_au.py
@@ -0,0 +1,22 @@
+from scrapy.linkextractors import LinkExtractor
+from scrapy.spiders import CrawlSpider, Rule
+
+from locations.google_url import extract_google_position
+from locations.spiders.vapestore_gb import clean_address
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class GuardianAUSpider(CrawlSpider, StructuredDataSpider):
+    name = "guardian_au"
+    item_attributes = {"brand": "Guardian", "brand_wikidata": "Q63367814"}
+    start_urls = ["https://www.guardianpharmacies.com.au/store-locator/"]
+    rules = [
+        Rule(LinkExtractor(allow=r"/store\-locator/\w{2,3}$")),
+        Rule(LinkExtractor(allow=r"/store\-locator/\w{2,3}/\w+"), callback="parse_sd"),
+    ]
+
+    def post_process_item(self, item, response, ld_data, **kwargs):
+        item["street_address"] = clean_address(ld_data["address"]["streetAddress"])
+        extract_google_position(item, response)
+
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Amcal (Australian pharmacy/chemist)
### Brand name

Amcal

### Wikidata ID

Q63367373

### Store finder url(s)

https://www.amcal.com.au/store-locator/vic
https://www.amcal.com.au/store-locator/nsw
https://www.amcal.com.au/store-locator/qld
https://www.amcal.com.au/store-locator/tas
https://www.amcal.com.au/store-locator/act
https://www.amcal.com.au/store-locator/wa
https://www.amcal.com.au/store-locator/sa
https://www.amcal.com.au/store-locator/nt


Discussion:

Resolving patch:
diff --git a/locations/spiders/amcal_au.py b/locations/spiders/amcal_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/amcal_au.py
@@ -0,0 +1,22 @@
+from scrapy.linkextractors import LinkExtractor
+from scrapy.spiders import CrawlSpider, Rule
+
+from locations.google_url import extract_google_position
+from locations.spiders.vapestore_gb import clean_address
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class AmcalAUSpider(CrawlSpider, StructuredDataSpider):
+    name = "amcal_au"
+    item_attributes = {"brand": "Amcal", "brand_wikidata": "Q63367373"}
+    start_urls = ["https://www.amcal.com.au/store-locator"]
+    rules = [
+        Rule(LinkExtractor(allow=r"/store\-locator/\w{2,3}$")),
+        Rule(LinkExtractor(allow=r"/store\-locator/\w{2,3}/\w+"), callback="parse_sd"),
+    ]
+
+    def post_process_item(self, item, response, ld_data, **kwargs):
+        item["street_address"] = clean_address(ld_data["address"]["streetAddress"])
+        extract_google_position(item, response)
+
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
New spider suggestion: Grape Tree (UK health food chain)
### Brand name

Grape Tree

### Wikidata ID

Q87071497

### Store finder url(s)

https://www.grapetree.co.uk/stores/locator/store/l

Note that https://www.grapetree.co.uk/sitemap.xml contains the stores but the URL paths there (which begin /store-locator/l/ ) are incorrect.


Discussion:

Resolving patch:
diff --git a/locations/spiders/grape_tree_gb.py b/locations/spiders/grape_tree_gb.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/grape_tree_gb.py
@@ -0,0 +1,7 @@
+from locations.storefinders.uberall import UberallSpider
+
+
+class GrapeTreeGBSpider(UberallSpider):
+    name = "grape_tree_gb"
+    item_attributes = {"brand": "Grape Tree", "brand_wikidata": "Q87071497"}
+    key = "N2WuycvqRWwy4Hd0bYpVzqN7jllSkA"


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Arctic Circle
https://acburger.com/menu/


Discussion:
Single JSON array of store locations and addresses: https://media.acburger.com/api/locations/json

I could not find opening hours listed on the website

Resolving patch:
diff --git a/locations/spiders/arctic_circle_us.py b/locations/spiders/arctic_circle_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/arctic_circle_us.py
@@ -0,0 +1,19 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+
+
+class ArcticCircleUSSpider(Spider):
+    name = "arctic_circle_us"
+    item_attributes = {"brand": "Arctic Circle", "brand_wikidata": "Q4787501"}
+    start_urls = ["https://media.acburger.com/api/locations/json"]
+
+    def parse(self, response, **kwargs):
+        for store in response.json():
+            if store["published"] != "1":
+                continue
+
+            store["ref"] = store.pop("store")
+            store["street_address"] = store.pop("address")
+
+            yield DictParser.parse(store)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Food City (AZ)
https://myfoodcity.com/location/


Discussion:

Resolving patch:
diff --git a/locations/spiders/food_city_us.py b/locations/spiders/food_city_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/food_city_us.py
@@ -0,0 +1,18 @@
+import html
+
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+
+
+class FoodCityUSSpider(Spider):
+    name = "food_city_us"
+    item_attributes = {"brand": "Food City", "brand_wikidata": "Q16981107"}
+    start_urls = ["https://myfoodcity.com/wp-admin/admin-ajax.php?action=store_search&max_results=250&autoload=1"]
+
+    def parse(self, response, **kwargs):
+        for location in response.json():
+            location["name"] = html.unescape(location.pop("store"))
+            location["street_address"] = ", ".join(filter(None, [location.pop("address"), location.pop("address2")]))
+
+            yield DictParser.parse(location)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Cousins Subs
https://www.cousinssubs.com/Order


Discussion:
URL for a JSON array of all store data: https://www.cousinssubs.com/api/restaurant/

Currently 94 stores are returned.

Resolving patch:
diff --git a/locations/spiders/cousins_subs_us.py b/locations/spiders/cousins_subs_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/cousins_subs_us.py
@@ -0,0 +1,16 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+
+
+class CousinsSubsUSSpider(Spider):
+    name = "cousins_subs_us"
+    item_attributes = {"brand": "Cousins Subs", "brand_wikidata": "Q5178843"}
+    start_urls = ["https://www.cousinssubs.com/api/restaurant/"]
+
+    def parse(self, response, **kwargs):
+        for location in response.json():
+            location["website"] = f'https://www.cousinssubs.com/order/{location["slug"]}'
+            location["street_address"] = location.pop("street")
+
+            yield DictParser.parse(location)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Yettel Serbia
### Brand name

Yettel

### Wikidata ID

Q1780171

### Store finder url(s)

https://www.yettel.rs/stores/latlong/

Mo - Ponedeljak
Tu - Utorak
We - Sreda
Th - Četvrtak
Fr - Petak
Sa - Subota
Su - Nedelja


Discussion:
Does 132 stores sound right? If so a single FormRequest should give all the data `curl -X POST https://www.yettel.rs/stores/latlong/ -d "lat=0&lng=0&dist=10000000000" `
It looks like there are 132 stores in the whole country. The IDs reach 350, but I assume that it's due to a lot of additions and removals in their DB.
Sounds good, do you want to write this one too? If not I can do it.
I will try to do Telenor Denmakr first and see what I can do.
OK, let me know if you need any help.

Resolving patch:
diff --git a/locations/hours.py b/locations/hours.py
--- a/locations/hours.py
+++ b/locations/hours.py
@@ -137,6 +137,16 @@
     "Søn": "Su",
 }
 
+DAYS_RS = {
+    "Ponedeljak": "Mo",
+    "Utorak": "Tu",
+    "Sreda": "We",
+    "Četvrtak": "Th",
+    "Petak": "Fr",
+    "Subota": "Sa",
+    "Nedelja": "Su",
+}
+
 
 def day_range(start_day, end_day):
     start_ix = DAYS.index(start_day)
diff --git a/locations/spiders/yettel_rs.py b/locations/spiders/yettel_rs.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/yettel_rs.py
@@ -0,0 +1,34 @@
+import scrapy
+
+from locations.hours import DAYS_RS, OpeningHours, sanitise_day
+from locations.items import Feature
+
+
+class YettelRSSpider(scrapy.Spider):
+    name = "yettel_rs"
+    item_attributes = {"brand": "Yettel", "brand_wikidata": "Q1780171"}
+
+    def start_requests(self):
+        url = "https://www.yettel.rs/stores/latlong/"
+        yield scrapy.http.FormRequest(url=url, formdata={"lat": "0", "lng": "0", "dist": "10000000000"}, method="POST")
+
+    def parse(self, response):
+        for index, store in enumerate(response.json()["data"]["stores"]):
+            item = Feature()
+            item["ref"] = store["id"]
+            item["lat"] = store["lat"]
+            item["lon"] = store["lng"]
+
+            item["city"] = store["city"]
+            item["postcode"] = store["post_number"]
+            item["street_address"] = store["address"]
+
+            oh = OpeningHours()
+
+            for wh in store["workingHoursFormated"]:
+                if day := sanitise_day(wh["day"], DAYS_RS):
+                    if wh.get("startTime") and wh.get("endTime"):
+                        oh.add_range(day, wh["startTime"], wh["endTime"])
+
+            item["opening_hours"] = oh
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Odeon Cinemas
### Brand name

Odeon Cinemas

### Wikidata ID

Q6127470

### Store finder url(s)

https://www.odeon.co.uk/sitemap.xml is the entire site map
https://www.odeon.co.uk/cinemas/*/ lists all the Odeon's.

\* = location


Discussion:

Resolving patch:
diff --git a/locations/spiders/odeon_gb.py b/locations/spiders/odeon_gb.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/odeon_gb.py
@@ -0,0 +1,30 @@
+import json
+
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+
+
+class OdeonGBSpider(Spider):
+    name = "odeon_gb"
+    item_attributes = {"brand": "Odeon", "brand_wikidata": "Q6127470"}
+    start_urls = ["https://www.odeon.co.uk/cinemas/"]
+    requires_proxy = True
+
+    def parse(self, response, **kwargs):
+        data = json.loads(response.xpath("//@data-v-site-list").get())
+        for location in data["config"]["cinemas"]:
+            location["address"] = ", ".join(
+                filter(
+                    None,
+                    [
+                        location.pop("addressLine1"),
+                        location.pop("addressLine2"),
+                        location.pop("addressLine3"),
+                        location.pop("addressLine4"),
+                        "United Kingdom",
+                    ],
+                )
+            )
+            location["url"] = f'https://www.odeon.co.uk{location["url"]}'
+            yield DictParser.parse(location)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Mitre 10 (New Zealand hardware store chain)
### Brand name

Mitre 10

### Wikidata ID

Q6882394

### Store finder url(s)

https://www.mitre10.co.nz/store-locator?branchId=X1&page=1&type=data

Paged JSON response needing a simple DictParser.parse()


Discussion:

Resolving patch:
diff --git a/locations/spiders/mitre_10_nz.py b/locations/spiders/mitre_10_nz.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/mitre_10_nz.py
@@ -0,0 +1,50 @@
+import json
+
+import scrapy
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_FULL, OpeningHours
+
+
+class Mitre10NZSpider(scrapy.Spider):
+    name = "mitre_10_nz"
+    item_attributes = {"brand": "Mitre 10", "brand_wikidata": "Q6882394"}
+    allowed_domains = ["www.mitre10.co.nz"]
+    start_urls = ["https://www.mitre10.co.nz/store-locator?branchId=X1&page=0&type=data"]
+
+    def parse(self, response):
+        current_page = response.meta.get("page", 0)
+        total_stores = response.json()["total"]
+        stores = response.json()["data"]
+        for store in stores:
+            item = DictParser.parse(store)
+            item["ref"] = store["storeCode"]
+            item["name"] = store["displayName"]
+            item.pop("website")
+            if len(store["storeLandingUrl"]) > 0:
+                if store["storeLandingUrl"][0] == "/":
+                    item["website"] = "https://www.mitre10.co.nz/store" + store["storeLandingUrl"]
+                else:
+                    item["website"] = store["storeLandingUrl"]
+            oh = OpeningHours()
+            for day in DAYS_FULL:
+                if day in store["openings"]:
+                    hours_raw = "".join(store["openings"][day].split())
+                elif day.upper() in store["openings"]:
+                    hours_raw = "".join(store["openings"][day.upper()].split())
+                else:
+                    continue
+                if hours_raw == "Closed":
+                    continue
+                open_time = hours_raw.split("-")[0]
+                close_time = hours_raw.split("-")[1]
+                oh.add_range(day, open_time, close_time, "%I:%M%p")
+            item["opening_hours"] = oh.as_opening_hours()
+            yield item
+        if current_page * 10 < total_stores - 10:
+            next_page = current_page + 1
+            yield scrapy.Request(
+                url=f"https://www.mitre10.co.nz/store-locator?branchId=X1&page={next_page}&type=data",
+                callback=self.parse,
+                meta={"page": next_page},
+            )


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Brumby's Bakeries (AU/NZ bakery chain)
### Brand name

Brumby's

### Wikidata ID

Q4978794

### Store finder url(s)

https://www.brumbys.com.au/store-locator/

There is an embedded <script> with a storesObj object that can be parsed. Individual store pages do not have opening hours, so parsing this storesObj object is most efficient without limiting data captured.


Discussion:

Resolving patch:
diff --git a/locations/spiders/brumbys_bakeries_au.py b/locations/spiders/brumbys_bakeries_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/brumbys_bakeries_au.py
@@ -0,0 +1,47 @@
+import json
+import re
+
+import chompjs
+import scrapy
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class BrumbysBakeriesAUSpider(scrapy.Spider):
+    name = "brumbys_bakeries_au"
+    item_attributes = {"brand": "Brumby's Bakeries", "brand_wikidata": "Q4978794"}
+    allowed_domains = ["www.brumbys.com.au"]
+    start_urls = ["https://www.brumbys.com.au/store-locator/"]
+
+    def parse(self, response):
+        data_raw = response.xpath('//script[@id="theme_app_js-js-extra"]/text()').get()
+        if m := re.search(r"var storesObj =\s*({.+});\s*var", data_raw):
+            stores = json.loads(chompjs.parse_js_object(m.group(1))["stores"])
+            for store in stores:
+                store["phone"] = store["phone"].split("tel:", 1)[1].split('"', 1)[0]
+                item = DictParser.parse(store)
+                item["lat"] = store["address"]["lat"]
+                item["lon"] = store["address"]["lng"]
+                item["addr_full"] = store["address"]["address"]
+                yield scrapy.Request(url=item["website"], callback=self.parse_hours, meta={"item": item})
+
+    def parse_hours(self, response):
+        item = response.meta["item"]
+        hours_raw = (
+            " ".join(
+                (
+                    " ".join(
+                        response.xpath(
+                            '//h3[contains(@class, "heading-hours")]/following::table/tbody/tr/td/text()'
+                        ).getall()
+                    )
+                ).split()
+            )
+            .upper()
+            .replace("24 HOURS", "Mon - Sun 12:00 AM - 11:59 PM")
+        )
+        oh = OpeningHours()
+        oh.add_ranges_from_string(hours_raw)
+        item["opening_hours"] = oh.as_opening_hours()
+        yield item


Test patch: