internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
`easypay_bg.py` - foreign offices are marked as located in Bulgaria
EasyPay seems to have some offices outside of Bulgaria (1 in Austria, 5 in Greece and 14 in Cyprus), but all of them have `addr:country=BG`. 


Discussion:
@Dimitar5555 As there are only some EasyPay offices outside Bulgaria, should we just hardcode the countries, or apply reverse geocoding of coordinates?
How would you hardcode the countries? There is no information in which country each office is. 

The best solution (that I can think of right now) would be to mark offices with postcode (4th column) with `addr:country=BG` and reverse geocode all other coordinates.
oh, ya that is good. Reverse geocode would have to run only 20 times.
I will implement this..
The pipeline already runs an offline reverse geocoder using [this code here](https://github.com/alltheplaces/alltheplaces/blob/master/locations/pipelines/country_code_clean_up.py), so it might be enough to set some properties on the spider.
Oh, I see. I will make the changes asap.

> EasyPay seems to have some offices outside of Bulgaria (1 in Austria, 5 in Greece and 14 in Cyprus), but all of them have `addr:country=BG`.

CORRECTION:
BG-3267
RO-24
CY-14
GR-10
MK-3
AT-1


Resolving patch:
diff --git a/locations/spiders/easypay_bg.py b/locations/spiders/easypay.py
similarity index 68%
rename from locations/spiders/easypay_bg.py
rename to locations/spiders/easypay.py
--- a/locations/spiders/easypay_bg.py
+++ b/locations/spiders/easypay.py
@@ -3,8 +3,8 @@
 from locations.dict_parser import DictParser
 
 
-class EasypayBGSpider(CSVFeedSpider):
-    name = "easypay_bg"
+class EasypaySpider(CSVFeedSpider):
+    name = "easypay"  # the name is not set to easypay_bg, because then name would be used to extract "country" by the pipeline, and "country" in every item would be set to "BG", which is wrong.
     item_attributes = {"brand": "EasyPay", "brand_wikidata": "Q110583289"}
     start_urls = ["https://www.easypay.bg/site/en/offices.csv"]
     delimiter = "|"


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
I&G Brokers
### Brand name

I&G Brokers

### Wikidata ID

Q110399829

### Store finder url(s)

https://iandgbrokers.com/offices/pins?srch_offices=&city=&features[]=all&_=1695470505675


Discussion:
Hey @Dimitar5555, I would like to work on this..

Hey @Arshad-Ali-1a, feel free to do so. 

Resolving patch:
diff --git a/locations/spiders/i_and_g_brokers.py b/locations/spiders/i_and_g_brokers.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/i_and_g_brokers.py
@@ -0,0 +1,21 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_BG, OpeningHours
+
+
+class IAndGBrokersSpider(Spider):
+    name = "i_and_g_brokers"
+    item_attributes = {"brand": "I&G Brokers", "brand_wikidata": "Q110399829"}
+    allowed_domains = ["iandgbrokers.com"]
+    start_urls = ["https://iandgbrokers.com/offices/pins?srch_offices=&city=&features[]=all&_=1695470505675"]
+
+    def parse(self, response):
+        for data in response.json()["markers"]["offices"]:
+            data = data["data"]
+            item = DictParser.parse(data)
+
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(data["worktime"], days=DAYS_BG)
+
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
DSV Locker parcel lockers (ZA)
### Brand name

DSV Locker

### Wikidata ID

Q1155771

### Store finder url(s)

https://dsvdirect.co.za/dsv-locker/#Locator (457 lockers)


Discussion:
A single API call to the following URL returns all location data as a JSON object:

`https://dsv-clientzone-map.afrigis.co.za/proxy/api/wfs/?layer=locker`
Currently 263 features are available to extract.
So, data of only 263 lockers has to be extracted?
I would love to implement this..

Resolving patch:
diff --git a/locations/spiders/dsv_locker_za.py b/locations/spiders/dsv_locker_za.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/dsv_locker_za.py
@@ -0,0 +1,30 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class DSVLockerZASpider(Spider):
+    name = "dsv_locker_za"
+    item_attributes = {"brand": "DSV Locker", "brand_wikidata": "Q1155771"}
+    allowed_domains = ["dsv-clientzone-map.afrigis.co.za"]
+    start_urls = ["https://dsv-clientzone-map.afrigis.co.za/proxy/api/wfs/?layer=locker"]
+
+    def parse(self, response):
+        for data in response.json()["result"]["features"]:
+            data = data["properties"]
+            item = DictParser.parse(data)
+            item["ref"] = data.get("location_code")
+            item["name"] = data.get("location_name")
+            item["lat"] = data.get("gps_latitude")
+            item["lon"] = data.get("gps_longitude")
+            item["phone"] = data.get("contact_dsv_tel")
+            item["email"] = data.get("contact_dsv_email")
+            item["city"] = data["suburb"]
+
+            item["opening_hours"] = OpeningHours()
+
+            if opening_hours := data.get("business_hours"):
+                item["opening_hours"].add_ranges_from_string(opening_hours.replace("24/7", "00:00-24:00"))
+
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
National Trust
### Brand name

National Trust

### Wikidata ID

Q333515

### Store finder url(s)

https://www.nationaltrust.org.uk/api/search/places?query=sw6%203er&placeSort=distance&lat=51.4672197&lon=-0.1925168&milesRadius=10000&pageStartIndex=0&pageSize=10000&lang=en&publicationChannel=NATIONAL_TRUST_ORG_UK&maxPlaceResults=1000&maxLocationPageResults=0
(returns 598 results)
or
https://www.nationaltrust.org.uk/sitemap.xml


Discussion:
Thanks! I've updated the API call to use a 10000mi radius and 10000 results returned. The API doesn't seem to mind if all results are returned in a single query.
hey @davidhicks and @jimbofreedman , can I work on this issue?

I'd be very happy - I wouldn't get around to it for a bit. Thank you!

Resolving patch:
diff --git a/locations/spiders/national_trust_gb.py b/locations/spiders/national_trust_gb.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/national_trust_gb.py
@@ -0,0 +1,21 @@
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class NationalTrustGBSpider(Spider):
+    name = "national_trust_gb"
+    item_attributes = {"brand": "National Trust", "brand_wikidata": "Q333515"}
+    allowed_domains = ["www.nationaltrust.org.uk"]
+    start_urls = [
+        "https://www.nationaltrust.org.uk/api/search/places?query=sw6%203er&placeSort=distance&lat=51.4672197&lon=-0.1925168&milesRadius=10000&pageStartIndex=0&pageSize=10000&lang=en&publicationChannel=NATIONAL_TRUST_ORG_UK&maxPlaceResults=1000&maxLocationPageResults=0"
+    ]
+
+    def parse(self, response):
+        for data in response.json()["pagedMultiMatch"]["results"]:
+            item = DictParser.parse(data)
+            item["ref"] = data["id"]["value"]
+            item["image"] = data["imageUrl"]
+            item["opening_hours"] = OpeningHours()
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
EG America brands (US) (convience stores/fuel stations) (1679 locations)
### Brand name

Kwik Shop, Loaf N' Jug, Cumberland Farms, Quik Stop, etc...

### Wikidata ID

Q6450417, Q6663398, Q1143685, Q105141709, etc...

### Store finder url(s)

https://www.kwikshop.com/api/stores-locator/store-locator-search/results?pageNumber=1
or
https://www.loafnjug.com/api/stores-locator/store-locator-search/results?pageNumber=1
or
...(probably other aliases per brand, but each seem to return all brand information)

3 results per page and I can't figure out how to increase it. There could be many requests needed for all locations.


Discussion:

Resolving patch:
diff --git a/locations/spiders/eg_america_us.py b/locations/spiders/eg_america_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/eg_america_us.py
@@ -0,0 +1,41 @@
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+
+
+class EGAmericaUSSpider(Spider):
+    name = "eg_america_us"
+    allowed_domains = ["www.cumberlandfarms.com"]
+    # The provided start_urls[0] returns all brand locations due to
+    # high/non-existent page number resulting in all locations being
+    # returned. The domain of start_urls[0] can be interchanged with
+    # domains belonging to other brands of EG America.
+    start_urls = ["https://www.cumberlandfarms.com/api/stores-locator/store-locator-search/results?pageNumber=10000"]
+    brands = {
+        1: {"brand": "Cumberland Farms", "brand_wikidata": "Q1143685"},
+        11: {"brand": "Quik Stop", "brand_wikidata": "Q105141709"},
+        12: {"brand": "TomThumb", "brand_wikidata": "Q123012206"},
+        13: {"brand": "Turkey Hill", "brand_wikidata": "Q42376970"},
+        # bannerid=14 is not used on any store locator for EG America
+        # brands however street level imagery appears to show these
+        # locations as Turkey Hill branded fuel stations.
+        14: {"brand": "Turkey Hill", "brand_wikidata": "Q42376970"},
+        15: {"brand": "Minit Mart", "brand_wikidata": "Q18154470"},
+        16: {"brand": "Fastrac", "brand_wikidata": "Q117324848"},
+        17: {"brand": "Certified Oil", "brand_wikidata": "Q100148356"},
+        18: {"brand": "Kwik Shop", "brand_wikidata": "Q6450417"},
+        19: {"brand": "Loaf N' Jug", "brand_wikidata": "Q6663398"},
+        20: {"brand": "Sprint", "brand_wikidata": "Q123012447"},
+    }
+
+    def start_requests(self):
+        yield JsonRequest(url=self.start_urls[0], callback=self.parse)
+
+    def parse(self, response):
+        for location in response.json()["value"]["mapResults"]:
+            item = DictParser.parse(location)
+            item.update(self.brands[location["bannerId"]])
+            item["street_address"] = item.pop("addr_full", None)
+            item["website"] = location.get("pageUrl")
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
QuickChek (US) (convenience stores) (109 locations)
### Brand name

QuickChek

### Wikidata ID

Q7271689

### Store finder url(s)

`curl 'https://quickchek.com/wp-admin/admin-ajax.php' -X POST -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'X-Requested-With: XMLHttpRequest' --data-raw 'action=get_sorted_locations&dist=300000&lat=40.554887&lng=-74.4642861'`


Discussion:

Resolving patch:
diff --git a/locations/spiders/quickchek_us.py b/locations/spiders/quickchek_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/quickchek_us.py
@@ -0,0 +1,30 @@
+from scrapy import FormRequest, Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS, OpeningHours
+
+
+class QuickChekUSSpider(Spider):
+    name = "quickchek_us"
+    item_attributes = {"brand": "QuickChek", "brand_wikidata": "Q7271689"}
+    allowed_domains = ["quickchek.com"]
+    start_urls = ["https://quickchek.com/wp-admin/admin-ajax.php"]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            headers = {
+                "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
+                "X-Requested-With": "XMLHttpRequest",
+            }
+            formdata = {"action": "get_sorted_locations", "dist": "100000", "lat": "0", "lng": "0"}
+            yield FormRequest(url=url, method="POST", formdata=formdata, headers=headers)
+
+    def parse(self, response):
+        for location in response.json():
+            item = DictParser.parse(location)
+            item["opening_hours"] = OpeningHours()
+            if location["hours"] == "Open 24 Hours":
+                item["opening_hours"].add_days_range(DAYS, "00:00", "23:59")
+            else:
+                item["opening_hours"].add_ranges_from_string("Mo-Su: {}".format(location["hours"]))
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Dirt Cheap
http://ilovedirtcheap.com/locations


Discussion:
Wikidata: Q123013019

Resolving patch:
diff --git a/locations/spiders/dirt_cheap_us.py b/locations/spiders/dirt_cheap_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/dirt_cheap_us.py
@@ -0,0 +1,12 @@
+from locations.storefinders.agile_store_locator import AgileStoreLocatorSpider
+
+
+class DirtCheapUSSpider(AgileStoreLocatorSpider):
+    name = "dirt_cheap_us"
+    item_attributes = {"brand": "Dirt Cheap", "brand_wikidata": "Q123013019"}
+    allowed_domains = ["ilovedirtcheap.com"]
+
+    def parse_item(self, item, location):
+        item["website"] = "https://ilovedirtcheap.com/locations/store-details/" + location["slug"]
+        item["image"] = location.get("storephoto")
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Evereve
https://www.evereve.com/storelocator.cfm


Discussion:

Resolving patch:
diff --git a/locations/spiders/evereve_us.py b/locations/spiders/evereve_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/evereve_us.py
@@ -0,0 +1,35 @@
+from html import unescape
+
+from scrapy import Request
+
+from locations.storefinders.amasty_store_locator import AmastyStoreLocatorSpider
+
+
+class EVEREVEUSSpider(AmastyStoreLocatorSpider):
+    name = "evereve_us"
+    item_attributes = {"brand": "EVEREVE", "brand_wikidata": "Q69891997"}
+    allowed_domains = ["evereve.com"]
+
+    def start_requests(self):
+        # The request won't work without the headers supplied below.
+        headers = {
+            "X-Requested-With": "XMLHttpRequest",
+        }
+        for domain in self.allowed_domains:
+            yield Request(url=f"https://{domain}/amlocator/index/ajax/", method="POST", headers=headers)
+
+    def parse_item(self, item, location, popup_html):
+        if "COMING SOON" in item["name"].upper():
+            return
+        popup_text = list(
+            filter(
+                None,
+                map(unescape, map(str.strip, popup_html.xpath('//div[@class="amlocator-info-popup"]/text()').getall())),
+            )
+        )
+        if len(popup_text) == 3:
+            item["addr_full"] = ", ".join([popup_text[0], popup_text[1]])
+            item["phone"] = popup_text[2]
+        elif len(popup_text) == 2:
+            item["addr_full"] = ", ".join([popup_text[0], popup_text[1]])
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Liberty (AU) (fuel stations) (92 locations)
### Brand name

Liberty

### Wikidata ID

Q106687078

### Store finder url(s)

https://www.libertyconvenience.com.au/wp-admin/admin-ajax.php?action=store_search&autoload=1


Discussion:
I'd like to work on this issue.

Resolving patch:
diff --git a/locations/spiders/liberty_au.py b/locations/spiders/liberty_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/liberty_au.py
@@ -0,0 +1,21 @@
+import scrapy
+from scrapy import Selector
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class LibertyAUSpider(scrapy.Spider):
+    name = "liberty_au"
+    item_attributes = {"brand": "Liberty Oil", "brand_wikidata": "Q106687078"}
+    allowed_domains = ["libertyconvenience.com.au"]
+    start_urls = ["https://www.libertyconvenience.com.au/wp-admin/admin-ajax.php?action=store_search&autoload=1"]
+
+    def parse(self, response):
+        for location in response.json():
+            item = DictParser.parse(location)
+            item["street_address"] = item.pop("addr_full", None)
+            hours_string = " ".join(filter(None, Selector(text=location["hours"]).xpath("//text()").getall()))
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(hours_string)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Blake's Lotaburger
https://en.wikipedia.org/wiki/Blake%27s_Lotaburger

Locations at https://www.lotaburger.com/locations/


Discussion:
All locations seemingly returned in JSON response for: `https://www.lotaburger.com/wp-json/ip/v1/blakes_location_json/`

Resolving patch:
diff --git a/locations/spiders/blakes_lotaburgers.py b/locations/spiders/blakes_lotaburgers.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/blakes_lotaburgers.py
@@ -0,0 +1,46 @@
+from scrapy import Spider
+
+from locations.categories import Extras, apply_yes_no
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class BlakesLotaburgersSpider(Spider):
+    name = "blakes_lotaburgers"
+    item_attributes = {"brand": "Blakes Lotaburgers", "brand_wikidata": "Q492430"}
+    allowed_domains = ["www.lotaburger.com"]
+    start_urls = ["https://www.lotaburger.com/wp-json/ip/v1/blakes_location_json/"]
+
+    def parse(self, response):
+        for data in response.json():
+            item = DictParser.parse(data)
+            item["ref"] = data["post_id"]
+            item["name"] = data["business_name"]
+            item["lat"] = data["coords"]["lat"]
+            item["lon"] = data["coords"]["long"]
+            item["street_address"] = data["business_address"]
+            item["city"] = data["business_city"]
+            item["state"] = data["business_state"]
+            item["postcode"] = data["business_zipcode"]
+            item["phone"] = data["business_phone"]
+            item["website"] = data["business_url"]
+            item["opening_hours"] = OpeningHours()
+            raw_string = "".join(
+                [
+                    opening_data["day"]
+                    + " "
+                    + opening_data["hours"][0]["from"]
+                    + "-"
+                    + opening_data["hours"][0]["to"]
+                    + "\n"
+                    for opening_data in data["store_hours"]
+                    if (
+                        type(opening_data["hours"][0]["from"]) is not bool
+                        and type(opening_data["hours"][0]["to"]) is not bool
+                    )
+                ]
+            )
+            item["opening_hours"].add_ranges_from_string(raw_string)
+            item["extras"]["breakfast"] = data["breakfast"]
+            apply_yes_no(Extras.DRIVE_THROUGH, item, len(data["drive_thru"]) > 0, False)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
M-texx clothes and shoes containers
### Brand name

M-texx

### Wikidata ID

Q122947768

### Store finder url(s)

https://m-texx.com/локации

It requires parsing of all `li` html elements. The four digit numbers are postcodes, there is no `ref` system.


Discussion:

Resolving patch:
diff --git a/locations/spiders/mtexx_bg.py b/locations/spiders/mtexx_bg.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/mtexx_bg.py
@@ -0,0 +1,37 @@
+import re
+
+from scrapy import Spider
+
+from locations.items import Feature
+
+
+class MtexxBGSpider(Spider):
+    name = "mtexx_bg"
+    item_attributes = {"brand": "M-texx", "brand_wikidata": "Q122947768"}
+    allowed_domains = ["www.m-texx.com"]
+    start_urls = ["https://m-texx.com/локации"]
+    no_refs = True
+
+    def parse(self, response):
+        for locations in response.xpath("//div[@data-ux='ContentText']"):
+            for location in locations.xpath(".//li/text()"):
+                text = location.get()
+                data = re.search(
+                    r'^(.+?)(\d+\.\d+|\d+°\d+\'\d+\.\d+"[NS])[\s,]+?(\d+\.\d+|\d+°\d+\'\d+\.\d+"[EW])', text
+                )
+                if data:
+                    properties = {
+                        "name": data.group(1),
+                        "lat": data.group(2),
+                        "lon": data.group(3),
+                    }
+                    if "N" in properties["lat"]:
+                        deg, minutes, seconds, direction = re.split("[°'\"]", properties["lat"])
+                        properties["lat"] = (float(deg) + float(minutes) / 60 + float(seconds) / (60 * 60)) * (
+                            -1 if direction == "S" else 1
+                        )
+                        deg, minutes, seconds, direction = re.split("[°'\"]", properties["lon"])
+                        properties["lon"] = (float(deg) + float(minutes) / 60 + float(seconds) / (60 * 60)) * (
+                            -1 if direction == "W" else 1
+                        )
+                    yield Feature(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Great Wolf Lodge
https://www.greatwolf.com/


Discussion:
CrawlSpider of `https://www.greatwolf.com/locations` with StructuredDataSpider used for individual location pages such as `https://www.greatwolf.com/mashantucket`. Coordinates or much other detail on each location does not appear to be available anywhere to parse.
It seems that there's coordinates on the customer service page for each location. For example: https://www.greatwolf.com/grand-mound/customer-service
Great, thanks!

In that case we should do: SitemapSpider of `https://www.greatwolf.com/php-root.sitemap.xml` with filter of `/customer-service$` and for each location page, use StructuredDataSpider to extract location data. Within the location data is a `hasMap` attribute which is a Google Maps URL that can be parsed to obtain coordinates e.g. `46.7905042,-123.0121885` for `https://www.greatwolf.com/grand-mound/customer-service`

Resolving patch:
diff --git a/locations/spiders/great_wolf_us.py b/locations/spiders/great_wolf_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/great_wolf_us.py
@@ -0,0 +1,23 @@
+from re import search
+from urllib.parse import unquote
+
+from scrapy.spiders import SitemapSpider
+
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class GreatWolfResortsUS(SitemapSpider, StructuredDataSpider):
+    name = "great_wolf_resorts_us"
+    item_attributes = {"brand": " Great Wolf Resorts", "brand_wikidata": "Q5600260"}
+    allowed_domains = ["www.greatwolf.com"]
+    sitemap_urls = ["https://www.greatwolf.com/php-root.sitemap.xml"]
+    sitemap_rules = [(r"/customer-service$", "parse_sd")]
+
+    def post_process_item(self, item, response, ld_data, **kwargs):
+        item.pop("facebook")
+        item.pop("image")
+        item.pop("twitter")
+        has_map = unquote(ld_data["hasMap"])
+        item["lat"], item["lon"] = search(r"[@=](-?\d+\.\d+),(-?\d+\.\d+)", has_map).groups()
+
+        yield item


Test patch: