internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
EmpRes
https://www.empres.com/locations/


Discussion:
All locations returned as JSON data with a single API request: `curl 'https://www.empres.com/wp-admin/admin-ajax.php' -X POST -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data-raw 'within=200&state=&service=&action=block_locations_get&position='`

Resolving patch:
diff --git a/locations/spiders/empres_us.py b/locations/spiders/empres_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/empres_us.py
@@ -0,0 +1,36 @@
+from scrapy import Request, Spider
+from scrapy.http import FormRequest
+
+from locations.dict_parser import DictParser
+from locations.user_agents import BROWSER_DEFAULT
+
+
+class EmpResUSSpider(Spider):
+    name = "empres_us"
+    item_attributes = {"brand": "EmpRes", "brand_wikidata": "Q123370276"}
+    allowed_domains = ["www.empres.com"]
+    start_urls = ["https://www.empres.com/wp-admin/admin-ajax.php"]
+    user_agent = BROWSER_DEFAULT
+
+    def start_requests(self):
+        formdata = {
+            "within": "200",
+            "state": "",
+            "service": "",
+            "action": "block_locations_get",
+            "position": "",
+        }
+        headers = {
+            "Origin": "https://www.empres.com",
+            "Referer": "https://www.empres.com/locations/",
+            "X-Requested-With": "XMLHttpRequest",
+        }
+        for url in self.start_urls:
+            yield FormRequest(url=url, formdata=formdata, headers=headers, method="POST")
+
+    def parse(self, response):
+        for location in response.json()["locations"]:
+            item = DictParser.parse(location)
+            item["website"] = location.get("link")
+            item["street_address"] = item.pop("street", None)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Versace
https://www.versace.com/us/en-us/find-a-store/#country=US&address=United+States&format=ajax


Discussion:
Probably better crawled through https://www.versace.com/sitemap-pages-international-en-1.xml with a URL filter of `https://www.versace.com/international/en/store/?StoreID=[A-Z]{2}\d{10}`

This returns a single page per store but XPath/CSS selectors would seemingly be needed (no structured data from what I can see).
All locations appear to be returned as JSON data via a single request to `https://www.versace.com/on/demandware.store/Sites-US-Site/en_US/Stores-Search`

Resolving patch:
diff --git a/locations/spiders/versace.py b/locations/spiders/versace.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/versace.py
@@ -0,0 +1,22 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+from locations.user_agents import BROWSER_DEFAULT
+
+
+class VersaceSpider(Spider):
+    name = "versace"
+    item_attributes = {"brand": "Versace", "brand_wikidata": "Q696376"}
+    allowed_domains = ["www.versace.com"]
+    start_urls = ["https://www.versace.com/on/demandware.store/Sites-US-Site/en_US/Stores-Search"]
+    user_agent = BROWSER_DEFAULT
+    custom_settings = {"ROBOTSTXT_OBEY": False}
+
+    def parse(self, response):
+        for location in response.json()["stores"]:
+            item = DictParser.parse(location)
+            item["street_address"] = ", ".join(filter(None, [location.get("address1"), location.get("address2")]))
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(location.get("storeHours"))
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Blain's Farm & Fleet 
https://www.farmandfleet.com/stores/


Discussion:
SitemapSpider of `https://www.farmandfleet.com/sitemap_stores.xml.gz` + StructuredDataSpider (LD+JSON, includes opening hours)

Resolving patch:
diff --git a/locations/spiders/blains_farm_and_fleet_us.py b/locations/spiders/blains_farm_and_fleet_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/blains_farm_and_fleet_us.py
@@ -0,0 +1,13 @@
+from scrapy.spiders import SitemapSpider
+
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class BlainsFarmAndFleetUSSpider(SitemapSpider, StructuredDataSpider):
+    name = "blains_farm_and_fleet_us"
+    item_attributes = {"brand": "Blain's Farm & Fleet", "brand_wikidata": "Q4923906"}
+    allowed_domains = ["www.farmandfleet.com"]
+    sitemap_urls = ["https://www.farmandfleet.com/sitemap_stores.xml.gz"]
+    sitemap_rules = [(r"\/stores\/[\w\-]+\/$", "parse_sd")]
+    time_format = "%H:%M:%S"
+    requires_proxy = True


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
HCR Manorcare
https://jobs.hcr-manorcare.com/locations


Discussion:
Renamed to ProMedica.

Single API query to `https://promedicaseniorcare.org/alllocations?formattedAddress=&boundsNorthEast=&boundsSouthWest=` seems to return all locations as XML data. JSON data is not returned even if headers are changed.

Resolving patch:
diff --git a/locations/spiders/promedica_us.py b/locations/spiders/promedica_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/promedica_us.py
@@ -0,0 +1,30 @@
+from scrapy.spiders import XMLFeedSpider
+
+from locations.items import Feature
+
+
+class ProMedicaUSSpider(XMLFeedSpider):
+    name = "promedica_us"
+    item_attributes = {"brand": "ProMedica", "brand_wikidata": "Q7246673"}
+    allowed_domains = ["promedicaseniorcare.org"]
+    start_urls = ["https://promedicaseniorcare.org/alllocations?formattedAddress=&boundsNorthEast=&boundsSouthWest="]
+    iterator = "xml"
+    itertag = "marker"
+
+    def parse_node(self, response, node):
+        properties = {
+            "ref": node.xpath(".//@web").get(),
+            "name": node.xpath(".//@name").get(),
+            "lat": node.xpath(".//@lat").get(),
+            "lon": node.xpath(".//@lng").get(),
+            "street_address": ", ".join(
+                filter(None, [node.xpath(".//@address").get(), node.xpath(".//@address2").get()])
+            ),
+            "city": node.xpath(".//@city").get(),
+            "state": node.xpath(".//@state").get(),
+            "postcode": node.xpath(".//@postal").get(),
+            "phone": node.xpath(".//@phone").get(),
+            "website": node.xpath(".//@web").get(),
+            "image": "https://promedicaseniorcare.org" + node.xpath(".//@image").get(),
+        }
+        yield Feature(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Touchstone Medical Imaging
https://www.touchstoneimaging.com/locations/


Discussion:
Storepoint spider with API key: `1632ba0abb55f7`

Resolving patch:
diff --git a/locations/spiders/touchstone_medical_imaging_us.py b/locations/spiders/touchstone_medical_imaging_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/touchstone_medical_imaging_us.py
@@ -0,0 +1,17 @@
+from locations.hours import DAYS_FULL, OpeningHours
+from locations.storefinders.storepoint import StorepointSpider
+
+
+class TouchstoneMedicalImagingUSSpider(StorepointSpider):
+    name = "touchstone_medical_imaging_us"
+    item_attributes = {"brand": "Touchstone Medical Imaging", "brand_wikidata": "Q123370518"}
+    key = "1632ba0abb55f7"
+
+    def parse_item(self, item, location):
+        item["opening_hours"] = OpeningHours()
+        hours_string = ""
+        for day_name in DAYS_FULL:
+            hours_range = location.get(day_name.lower(), "")
+            hours_string = f"{hours_string} {day_name}: {hours_range}"
+        item["opening_hours"].add_ranges_from_string(hours_string)
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
iFly
https://www.iflyworld.com/find-a-location/


Discussion:
All locations appear to be returned as JSON data by API call:
`curl 'https://dataanywhereprod.azure-api.net/SL/V2/api/stores/' -H 'Accept: application/json' -H 'Referer: https://www.iflyworld.com/' -H 'ocp-apim-subscription-key: d8c538d6a8de45398f6f1d4a6405102f' -H 'Origin: https://www.iflyworld.com'`

Resolving patch:
diff --git a/locations/spiders/ifly_us.py b/locations/spiders/ifly_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/ifly_us.py
@@ -0,0 +1,49 @@
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class iFLYUSSpider(Spider):
+    name = "ifly_us"
+    item_attributes = {"brand": "iFLY", "brand_wikidata": "Q64767105"}
+    allowed_domains = ["dataanywhereprod.azure-api.net"]
+    start_urls = ["https://dataanywhereprod.azure-api.net/SL/V2/api/stores/"]
+
+    def start_requests(self):
+        headers = {
+            "Referer": "https://www.iflyworld.com/",
+            "ocp-apim-subscription-key": "d8c538d6a8de45398f6f1d4a6405102f",
+            "Origin": "https://www.iflyworld.com",
+        }
+        for url in self.start_urls:
+            yield JsonRequest(url=url, headers=headers)
+
+    def parse(self, response):
+        for location in response.json()["StoreLocations"]:
+            item = DictParser.parse(location)
+            item["ref"] = location["ExtraData"]["ReferenceCode"]
+            item["geometry"] = location["Location"]
+            item["street_address"] = ", ".join(
+                filter(
+                    None,
+                    [
+                        location["ExtraData"]["Address"].get("AddressNonStruct_Line1"),
+                        location["ExtraData"]["Address"].get("AddressNonStruct_Line2"),
+                    ],
+                )
+            )
+            item["city"] = location["ExtraData"]["Address"].get("Locality")
+            item["state"] = location["ExtraData"]["Address"].get("Region")
+            item["postcode"] = location["ExtraData"]["Address"].get("PostalCode")
+            item["phone"] = location["ExtraData"].get("Phone")
+            item["opening_hours"] = OpeningHours()
+            for day_abbrev, hours_ranges in location["ExtraData"]["HoursOfOpStruct"].items():
+                if day_abbrev == "SpecialHours" or not hours_ranges.get("Ranges"):
+                    continue
+                for hours_range in hours_ranges["Ranges"]:
+                    item["opening_hours"].add_range(
+                        day_abbrev.title(), hours_range["StartTime"], hours_range["EndTime"], "%I:%M%p"
+                    )
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Battery World (AU) (110 locations)
### Brand name

Battery World

### Wikidata ID

Q124003678

### Store finder url(s)

https://www.batteryworld.com.au/app/services/Store.Service.ss


Discussion:

Resolving patch:
diff --git a/locations/spiders/battery_world_au.py b/locations/spiders/battery_world_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/battery_world_au.py
@@ -0,0 +1,28 @@
+from scrapy import Selector, Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class BatteryWorldAUSpider(Spider):
+    name = "battery_world_au"
+    item_attributes = {"brand": "Battery World", "brand_wikidata": "Q124003678"}
+    allowed_domains = ["www.batteryworld.com.au"]
+    start_urls = ["https://www.batteryworld.com.au/app/services/Store.Service.ss"]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            yield JsonRequest(url=url)
+
+    def parse(self, response):
+        for location in response.json():
+            item = DictParser.parse(location)
+            item["ref"] = location.get("internalid")
+            item["street_address"] = ", ".join(filter(None, [location.get("address1"), location.get("address2")]))
+            item["website"] = "https://www.batteryworld.com.au/stores/" + location.get("urlcomponent")
+            hours_html = Selector(text=location.get("openingHours"))
+            hours_string = " ".join(hours_html.xpath(".//text()").getall())
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(hours_string)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Clark Rubber (AU) (60 locations)
### Brand name

Clark Rubber

### Wikidata ID

Q124003720

### Store finder url(s)

https://www.clarkrubber.com.au/files/maps/locations.json


Discussion:

Resolving patch:
diff --git a/locations/spiders/clark_rubber_au.py b/locations/spiders/clark_rubber_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/clark_rubber_au.py
@@ -0,0 +1,26 @@
+from scrapy import Selector, Spider
+
+from locations.hours import OpeningHours
+from locations.items import Feature
+
+
+class ClarkRubberAUSpider(Spider):
+    name = "clark_rubber_au"
+    item_attributes = {"brand": "Clark Rubber", "brand_wikidata": "Q124003720"}
+    allowed_domains = ["www.clarkrubber.com.au"]
+    start_urls = ["https://www.clarkrubber.com.au/files/maps/locations.json"]
+
+    def parse(self, response):
+        for location in response.json()["features"]:
+            properties = {
+                "ref": location["properties"].get("siteID"),
+                "name": location["properties"].get("name"),
+                "geometry": location["geometry"],
+                "addr_full": location["properties"].get("address"),
+                "phone": location["properties"].get("phone"),
+            }
+            hours_html = Selector(text=location["properties"].get("hours"))
+            hours_text = " ".join(hours_html.xpath("//text()").getall())
+            properties["opening_hours"] = OpeningHours()
+            properties["opening_hours"].add_ranges_from_string(hours_text)
+            yield Feature(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Brass Tap Beer Bar
https://www.brasstapbeerbar.com/locations.aspx


Discussion:
All store locations appear to be returned as JSON data with a single call to `https://www.brasstapbeerbar.com/pinsNearestBrassTap.ashx?lat1=27.950&lon1=-82.45&range=100000&food=%25&lunch=%25&brunch=%25&music=%25`

Resolving patch:
diff --git a/locations/spiders/the_brass_tap_us.py b/locations/spiders/the_brass_tap_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/the_brass_tap_us.py
@@ -0,0 +1,35 @@
+from scrapy import Request, Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class TheBrassTapUSSpider(Spider):
+    name = "the_brass_tap_us"
+    item_attributes = {"brand": "The Brass Tap", "brand_wikidata": "Q64358002"}
+    allowed_domains = ["www.brasstapbeerbar.com"]
+    start_urls = [
+        "https://www.brasstapbeerbar.com/pinsNearestBrassTap.ashx?lat1=27.950&lon1=-82.45&range=100000&food=%25&lunch=%25&brunch=%25&music=%25"
+    ]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            yield JsonRequest(url=url)
+
+    def parse(self, response):
+        for location in response.json():
+            if location.get("comingsoon") and location.get("comingsoon") == "X":
+                continue
+            item = DictParser.parse(location)
+            item["street_address"] = item.pop("addr_full", None)
+            item["postcode"] = item.get("postcode").strip()
+            item["website"] = "https://www.brasstapbeerbar.com/" + location.get("urltag")
+            yield Request(url=item["website"], meta={"item": item}, callback=self.add_hours)
+
+    def add_hours(self, response):
+        item = response.meta["item"]
+        hours_string = " ".join(filter(None, response.xpath('//div[@id="MainContent_divHours"]//text()').getall()))
+        item["opening_hours"] = OpeningHours()
+        item["opening_hours"].add_ranges_from_string(hours_string)
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
bupa_gb spider is including closed branches
The bupa_gb.py spider is returning a number of closed practices, as these still have web pages and are still listed in the sitemap. Current examples include:

https://www.bupa.co.uk/dental/dental-care/practices/harleston
https://www.bupa.co.uk/dental/dental-care/practices/leckhampton

I think these can be reliably detected by checking if the name ends (case-insensitively) with "closed". So I'd suggest we drop any whose name matches /closed$/i. There are about 12 of these in total out of 388 branches.

(Sorry, I can no longer run the code on my computer to implement and test this myself, since I'm unable to install a recent enough version of Python.)


Discussion:

Resolving patch:
diff --git a/locations/items.py b/locations/items.py
--- a/locations/items.py
+++ b/locations/items.py
@@ -2,6 +2,7 @@
 #
 # See documentation in:
 # http://doc.scrapy.org/en/latest/topics/items.html
+from datetime import datetime
 
 import scrapy
 
@@ -78,3 +79,7 @@ def add_social_media(item: Feature, service: str, account: str):
         item[service] = account
     else:
         item["extras"][f"contact:{service}"] = account
+
+
+def set_closed(item: Feature, end_date: datetime = None):
+    item["extras"]["end_date"] = end_date.strftime("%Y-%m-%d") if end_date else "yes"
diff --git a/locations/spiders/bupa_gb.py b/locations/spiders/bupa_gb.py
--- a/locations/spiders/bupa_gb.py
+++ b/locations/spiders/bupa_gb.py
@@ -1,6 +1,7 @@
 from scrapy.spiders import SitemapSpider
 
 from locations.categories import Categories
+from locations.items import set_closed
 from locations.structured_data_spider import StructuredDataSpider
 
 
@@ -13,4 +14,8 @@ class BupaGBSpider(SitemapSpider, StructuredDataSpider):
     def post_process_item(self, item, response, ld_data, **kwargs):
         if "Total Dental Care" in item["name"]:
             item["brand"] = "Total Dental Care"
+
+        if item["name"].lower().endswith(" - closed"):
+            set_closed(item)
+
         yield item


Test patch:
diff --git a/tests/test_closed.py b/tests/test_closed.py
new file mode 100644
--- /dev/null
+++ b/tests/test_closed.py
@@ -0,0 +1,17 @@
+from datetime import datetime
+
+from locations.items import Feature, set_closed
+
+
+def test_unknown():
+    item = Feature()
+    set_closed(item)
+    assert item["extras"]["end_date"]
+    assert item["extras"]["end_date"] == "yes"
+
+
+def test_date():
+    item = Feature()
+    set_closed(item, datetime(2023, 12, 31))
+    assert item["extras"]["end_date"]
+    assert item["extras"]["end_date"] == "2023-12-31"
Repository: alltheplaces/alltheplaces

Issue:
Pague Menos (Brazilian pharmacy/chemist)
Not sure how this works, but let me start with a big pharmacy company in Brazil, with over 1000+ stores. Complete data, with addresses (addr:street, addr:housenumber, addr:postcode, addr:city and addr:state) and phone number. Each location has a single URL too.

### Brand name

Pague Menos

### Wikidata ID

Q7124466

### Store finder url(s)

https://www.paguemenos.com.br/nossas-lojas


Discussion:
Unfortunately they don't have coords, addresses, phone numbers and urls are available though. [Example data](https://pmenos.paguemenos.com.br/wp-json/wp/v2/lojas?per_page=1&page=1&order=asc)

Resolving patch:
diff --git a/locations/spiders/pague_menos_br.py b/locations/spiders/pague_menos_br.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/pague_menos_br.py
@@ -0,0 +1,36 @@
+import json
+from typing import Any
+
+from scrapy import Spider
+from scrapy.http import JsonRequest, Response
+
+from locations.items import Feature
+from locations.settings import DEFAULT_PLAYWRIGHT_SETTINGS
+
+
+class PagueMenosBRSpider(Spider):
+    name = "pague_menos_br"
+    item_attributes = {"brand": "Pague Menos", "brand_wikidata": "Q7124466"}
+    start_urls = ["https://www.paguemenos.com.br/nossas-lojas"]
+    is_playwright_spider = True
+    custom_settings = DEFAULT_PLAYWRIGHT_SETTINGS
+
+    def parse(self, response: Response, **kwargs: Any) -> Any:
+        # Collect cookies
+        yield JsonRequest(
+            "https://pmenos.paguemenos.com.br/wp-json/wp/v2/lojas?per_page=9999&page=1&order=asc",
+            callback=self.parse_api,
+        )
+
+    def parse_api(self, response: Response, **kwargs: Any) -> Any:
+        for location in json.loads(response.xpath("//text()").get()):
+            item = Feature()
+            item["ref"] = location["meta_box"]["id_loja"]
+            item["website"] = location["link"]
+            item["phone"] = location["meta_box"]["telefone"].split("/")[0]
+            item["street_address"] = location["meta_box"]["endereco"]
+            item["city"] = location["meta_box"]["cidade"]
+            item["state"] = location["meta_box"]["uf"]
+            item["postcode"] = location["meta_box"]["cep"]
+
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Caffe Nero GB spider using outdated JSON file
The caffe_nero_gb.py spider gets its data from JSON file that the Store Finder page at https://caffenero.com/uk/stores/ uses to display its map. However, it looks like that URL of that JSON file has changed, and ATP is still referencing the old (and no longer updated one).

The ATP code currently has
`allowed_domains = ["caffenero-webassets-production.s3.eu-west-2.amazonaws.com"]`
`start_urls = ["https://caffenero-webassets-production.s3.eu-west-2.amazonaws.com/stores/stores_gb.json"]`
But the URL referenced by https://caffenero.com/uk/stores/ is now
https://caffenerowebsite.blob.core.windows.net/production/data/stores/stores-gb.json

I think the format of the JSON file has remained the same, so it should just be a matter of swapping the URLs over.

To help issues like this be picked up sooner in the future, I wonder if there's a way of checking that the JSON URL used is still included in the https://caffenero.com/uk/stores/ page, and producing a warning to anyone running ATP if not?


Discussion:

Resolving patch:
diff --git a/locations/spiders/caffe_nero_gb.py b/locations/spiders/caffe_nero_gb.py
--- a/locations/spiders/caffe_nero_gb.py
+++ b/locations/spiders/caffe_nero_gb.py
@@ -1,5 +1,8 @@
+import re
+from typing import Any
+
 from scrapy import Spider
-from scrapy.http import JsonRequest
+from scrapy.http import JsonRequest, Response
 
 from locations.categories import Categories, Extras, apply_category, apply_yes_no
 from locations.dict_parser import DictParser
@@ -9,14 +12,15 @@
 class CaffeNeroGBSpider(Spider):
     name = "caffe_nero_gb"
     item_attributes = {"brand": "Caffe Nero", "brand_wikidata": "Q675808"}
-    allowed_domains = ["caffenero-webassets-production.s3.eu-west-2.amazonaws.com"]
-    start_urls = ["https://caffenero-webassets-production.s3.eu-west-2.amazonaws.com/stores/stores_gb.json"]
+    allowed_domains = ["caffenero.com", "caffenerowebsite.blob.core.windows.net"]
+    start_urls = ["https://caffenero.com/uk/stores/"]
 
-    def start_requests(self):
-        for url in self.start_urls:
-            yield JsonRequest(url=url)
+    def parse(self, response: Response, **kwargs: Any) -> Any:
+        yield JsonRequest(
+            re.search(r"loadGeoJson\(\n\s+'(https://.+)', {", response.text).group(1), callback=self.parse_geojson
+        )
 
-    def parse(self, response):
+    def parse_geojson(self, response: Response, **kwargs: Any) -> Any:
         for location in response.json()["features"]:
             if (
                 not location["properties"]["status"]["open"]
@@ -30,6 +34,8 @@ def parse(self, response):
             if location["properties"]["status"]["express"]:
                 item["brand"] = "Nero Express"
 
+            item["branch"] = item.pop("name")
+
             item["opening_hours"] = OpeningHours()
             for day_name, day_hours in location["properties"]["hoursRegular"].items():
                 if day_hours["open"] == "closed" or day_hours["close"] == "closed":


Test patch: