internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
Lilly Drogeries
### Brand name

Lilly

### Wikidata ID

Q111764460

### Store finder url(s)

Greece: https://lilidrogerie.gr/wp-json/v1/get-store/?lang=el
Serbia: https://www.lilly.rs/phpsqlsearch_genjson.php?lat=44.8019&lng=20.4671&radius=600&limit=200&keyword=none
Bulgaria: https://www.lillydrogerie.bg/bg/stores (look for `var locations = ` in the page source code, requires scraping HTML data in order to get opening hours)

It may be a good idea to have a separate spider for each country since all of them use diffrent systems and we don't know when (and if) they will standartise it.


Discussion:

Resolving patch:
diff --git a/locations/spiders/lilly_bg.py b/locations/spiders/lilly_bg.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/lilly_bg.py
@@ -0,0 +1,37 @@
+import re
+from html import unescape
+
+from chompjs import parse_js_object
+from scrapy import Selector, Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_BG, OpeningHours
+
+
+class LillyBGSpider(Spider):
+    name = "lilly_bg"
+    item_attributes = {"brand": "Lilly", "brand_wikidata": "Q111764460"}
+    allowed_domains = ["www.lillydrogerie.bg"]
+    start_urls = ["https://www.lillydrogerie.bg/bg/stores"]
+
+    def parse(self, response):
+        js_blob = response.xpath('//script[contains(text(), "var locations = {")]/text()').get()
+        js_blob = js_blob.split("var locations = ", 1)[1].split("}};", 1)[0] + "}}"
+        for location_id, location in parse_js_object(js_blob).items():
+            location["latitude"] = location["latitude"].replace(",", "")
+            location["longitude"] = location["longitude"].replace(",", "")
+            item = DictParser.parse(location)
+            item["ref"] = location_id
+            desc_html = Selector(text=location["description"])
+            item["addr_full"] = re.sub(
+                r"\s+",
+                " ",
+                unescape(desc_html.xpath("//p[1]/text()").get("").replace("Адрес:", "").replace("\\xA0", " ")),
+            ).strip()
+            item["phone"] = unescape(
+                desc_html.xpath('//p[contains(text(), "Тел:")]/text()').get("").replace("Тел: +", "")
+            )
+            hours_string = " ".join(filter(None, desc_html.xpath("//p/text()").getall()))
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(hours_string, DAYS_BG)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
A1 Austria
### Brand name

A1

### Wikidata ID

Q688755

### Store finder url(s)

All coordinates are located in https://www.a1.net/o/gucci/widgets/apis/shopfinder/bff/microservice-shopfinder/shop/basicData. There are several types of shops (dealer, post, it partner, shop, preferred partner, exclusive store). I'm not sure which ones we need.

The opening hours can be requested using `https://www.a1.net/o/gucci/widgets/apis/shopfinder/bff/microservice-shopfinder/shop/detail/id/0000028600` where the number is the shop id.


Discussion:

Resolving patch:
diff --git a/locations/spiders/a1_at.py b/locations/spiders/a1_at.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/a1_at.py
@@ -0,0 +1,43 @@
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_AT, OpeningHours
+
+
+class A1ATSpider(Spider):
+    name = "a1_at"
+    item_attributes = {"brand": "A1", "brand_wikidata": "Q688755"}
+    allowed_domains = ["www.a1.net"]
+    start_urls = ["https://www.a1.net/o/gucci/widgets/apis/shopfinder/bff/microservice-shopfinder/shop/basicData"]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            yield JsonRequest(url=url, callback=self.parse_store_list)
+
+    def parse_store_list(self, response):
+        for location in response.json():
+            if location["type"] != "SHOP":
+                # DEALER, IT_PARTNER, POST, EXCLUSIVE_STORE
+                # and PREFERRED_PARTNER should be ignored.
+                continue
+            item = DictParser.parse(location)
+            item["street_address"] = item.pop("addr_full", None)
+            yield JsonRequest(
+                url=f'https://www.a1.net/o/gucci/widgets/apis/shopfinder/bff/microservice-shopfinder/shop/detail/id/{item["ref"]}',
+                meta={"item": item},
+                callback=self.parse_store,
+            )
+
+    def parse_store(self, response):
+        item = response.meta["item"]
+        location = response.json()["basicShopData"]
+        item["phone"] = location.get("phone")
+        item["email"] = location.get("email")
+        item["website"] = location.get("url")
+        item["opening_hours"] = OpeningHours()
+        for day_hours in response.json().get("hoursShops", []):
+            item["opening_hours"].add_range(
+                DAYS_AT[day_hours["days"]], day_hours["tsFrom"], day_hours["tsTo"], "%H%M%S"
+            )
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
A1 Croatia
### Brand name

Q688755

### Wikidata ID

A1

### Store finder url(s)

https://www.a1.hr/prodajna-mjesta?p_p_id=pos_WAR_pos&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_resource_id=searchPos&p_p_cacheability=cacheLevelPage

There is data that needs to be sent via a POST request to the same url (`query=&tipprikaza=lista&types=center&types=partner`), otherwise no results are returned. I'm not sure how their partners operate and whether there are actual A1 stores there or the partners offer limited A1 services.


Discussion:
`curl 'https://www.a1.hr/prodajna-mjesta?p_p_id=pos_WAR_pos&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_resource_id=searchPos&p_p_cacheability=cacheLevelPage' -X POST -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data-raw 'query=&tipprikaza=lista&types=center&types=partner'`

From the results, only `type="center"` should be extracted.

Resolving patch:
diff --git a/locations/spiders/a1_hr.py b/locations/spiders/a1_hr.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/a1_hr.py
@@ -0,0 +1,42 @@
+from scrapy import Request, Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_EN, OpeningHours
+
+
+class A1HRSpider(Spider):
+    name = "a1_hr"
+    item_attributes = {"brand": "A1", "brand_wikidata": "Q688755"}
+    allowed_domains = ["www.a1.hr"]
+    start_urls = [
+        "https://www.a1.hr/prodajna-mjesta?p_p_id=pos_WAR_pos&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_resource_id=searchPos&p_p_cacheability=cacheLevelPage"
+    ]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            yield Request(
+                url=url,
+                body="query=&tipprikaza=lista&types=center&types=partner",
+                headers={"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"},
+                method="POST",
+            )
+
+    def parse(self, response):
+        for location in response.json()["result"]:
+            if location["type"] != "center":
+                # type="partner" should be ignored.
+                continue
+            item = DictParser.parse(location)
+            if location.get("branchId"):
+                item["ref"] = location["branchId"]
+            item["name"] = location["location"]
+            item["street_address"] = item.pop("addr_full", None)
+            item["opening_hours"] = OpeningHours()
+            for day_name, hours_range in location["workHours"].items():
+                if day_name.title() not in DAYS_EN.keys():
+                    continue
+                if hours_range == "-":
+                    # Closed
+                    continue
+                item["opening_hours"].add_range(DAYS_EN[day_name.title()], *hours_range.split("-", 1), "%H:%M")
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
A1 North Macedonia
### Brand name

A1

### Wikidata ID

Q24589907

### Store finder url(s)

https://www.a1.mk/poddrska/prodazni-mesta

There is a `select` element with `id=select-city`. From there we take all values and send them one by one to `https://www.a1.mk/o/StoreFinder-portlet/rest/api/shops/city/{city_id}`. They have a very nice machine readable format for opening hours and they even have `workingDuringHolidays`.


Discussion:
I think you can obtain a list of all shops (including opening hours) via:

`https://www.a1.mk/o/StoreFinder-portlet/rest/api/shops/`

80 results returned. Is that all?
I think so. A1 is quite rare to find even in Bulgaria. There are very few shops outside of the top 4 cities by population.

Resolving patch:
diff --git a/locations/spiders/a1_mk.py b/locations/spiders/a1_mk.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/a1_mk.py
@@ -0,0 +1,44 @@
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class A1MKSpider(Spider):
+    name = "a1_mk"
+    item_attributes = {"brand": "A1", "brand_wikidata": "Q24589907"}
+    allowed_domains = ["www.a1.mk"]
+    start_urls = ["https://www.a1.mk/o/StoreFinder-portlet/rest/api/shops/"]
+
+    def start_requests(self):
+        for url in self.start_urls:
+            yield JsonRequest(url=url)
+
+    def parse(self, response):
+        for location in response.json():
+            if not location.get("isOpen"):
+                continue
+            item = DictParser.parse(location)
+            if location["shopType"]["shopGroup"]["metadata"] != "centers":
+                # Ignore partner stores.
+                continue
+            if location.get("code"):
+                item["ref"] = location["code"]
+            item["name"] = location.get("nameMK")
+            item["extras"]["name:mk"] = location.get("nameMK")
+            item["extras"]["name:en"] = location.get("nameEN")
+            item["extras"]["name:al"] = location.get("nameAL")
+            item["street_address"] = location.get("addressMK")
+            item["city"] = location["city"].get("nameMK")
+            item["opening_hours"] = OpeningHours()
+            if location.get("workingWeekWorkPeriod") and location["workingWeekWorkPeriod"].strip() != "-":
+                item["opening_hours"].add_days_range(
+                    ["Mo", "Tu", "We", "Th", "Fr"], *location["workingWeekWorkPeriod"].split(" - "), "%H:%M"
+                )
+            if location.get("saturdayWorkingPeriod") and location["saturdayWorkingPeriod"].strip() != "-":
+                item["opening_hours"].add_range("Sa", *location["saturdayWorkingPeriod"].split(" - "), "%H:%M")
+            if location.get("sundayWorkingPeriod") and location["sundayWorkingPeriod"].strip() != "-":
+                item["opening_hours"].add_range("Su", *location["sundayWorkingPeriod"].split(" - "), "%H:%M")
+            yield item
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
A1 Serbia
### Brand name

A1

### Wikidata ID

Q826186

### Store finder url(s)

https://a1.rs/o-a1/prodaja/prodajna_mesta_mapa

There is a variable called `cities_json`. From it we need the city id only, which is passed to `https://a1.rs/sr_latin/api/pos/get_cities?city_id={city_id}`. 


Discussion:

Resolving patch:
diff --git a/locations/hours.py b/locations/hours.py
--- a/locations/hours.py
+++ b/locations/hours.py
@@ -405,8 +405,9 @@
 
 DAYS_SR = {
     "Pon": "Mo",
-    "Ponedjeljak": "Mo",
+    "Ponedeljak": "Mo",
     "Ponediljak": "Mo",
+    "Ponedjeljak": "Mo",
     "Понедељак": "Mo",
     "Понеделник": "Mo",
     "Uto": "Tu",
diff --git a/locations/spiders/a1_rs.py b/locations/spiders/a1_rs.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/a1_rs.py
@@ -0,0 +1,38 @@
+from chompjs import parse_js_object
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_SR, OpeningHours
+
+
+class A1RSSpider(Spider):
+    name = "a1_rs"
+    item_attributes = {"brand": "A1", "brand_wikidata": "Q826186"}
+    allowed_domains = ["a1.rs"]
+    start_urls = ["https://a1.rs/o-a1/prodaja/prodajna_mesta_mapa"]
+
+    def parse(self, response):
+        cities_js = (
+            response.xpath('//script[contains(text(), "var cities_json = ")]/text()')
+            .get()
+            .split("var cities_json = ", 1)[1]
+            .split("}];", 1)[0]
+            + "}]"
+        )
+        cities_dict = parse_js_object(cities_js)
+        for city in cities_dict:
+            yield JsonRequest(
+                url=f'https://a1.rs/sr_latin/api/pos/get_cities?city_id={city["id"]}', callback=self.parse_city_list
+            )
+
+    def parse_city_list(self, response):
+        for location in response.json():
+            if location["name"] != "A1 Centar":
+                # Ignore partner stores.
+                continue
+            item = DictParser.parse(location)
+            item["street_address"] = item.pop("addr_full", None)
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(location["working_hours"], days=DAYS_SR)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
A1 Slovenia
### Brand name

A1

### Wikidata ID

Q68755

### Store finder url(s)

https://www.a1.si/pomoc-in-informacije/prodajna-mesta

All the data is located in `shopBranchList`. It seems like A1 stores have `type=0` and `type=1`.


Discussion:

Resolving patch:
diff --git a/locations/spiders/a1_si.py b/locations/spiders/a1_si.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/a1_si.py
@@ -0,0 +1,39 @@
+from chompjs import parse_js_object
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_SI, OpeningHours
+
+
+class A1SISpider(Spider):
+    name = "a1_si"
+    item_attributes = {"brand": "A1", "brand_wikidata": "Q68755"}
+    allowed_domains = ["www.a1.si"]
+    start_urls = ["https://www.a1.si/pomoc-in-informacije/prodajna-mesta"]
+
+    def parse(self, response):
+        locations_js = (
+            response.xpath('//script[contains(text(), "var shopBranchList = ")]/text()')
+            .get()
+            .split("var shopBranchList = ", 1)[1]
+            .split("var savedFilters=[];", 1)[0]
+        )
+        locations = parse_js_object(locations_js)
+        for location in locations:
+            if location["type"] != 0 and location["type"] != 1:
+                continue
+            if location["openStatus"] != "open":
+                continue
+            item = DictParser.parse(location)
+            if not location.get("id"):
+                # Numerous store IDs are missing. Phone numbers
+                # appear to be unique per store so these can be used
+                # instead where a store ID is missing.
+                item["ref"] = location.get("phone")
+            item["name"] = location.get("nameOfShop")
+            item["lat"] = location.get("latitude", "").replace(",", ".")
+            item["lon"] = location.get("longitude", "").replace(",", ".")
+            item["street_address"] = item.pop("addr_full", None)
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(location["workingHours"], days=DAYS_SI)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Bealls
https://stores.beallsflorida.com/search.html


Discussion:
Single API call returns all location data as JSON data: `https://maps.stores.beallsflorida.com/api/getAsyncLocations?template=search&level=search&search=FL,%20USA&radius=10000&limit=10000`

Resolving patch:
diff --git a/locations/spiders/bealls_florida_us.py b/locations/spiders/bealls_florida_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/bealls_florida_us.py
@@ -0,0 +1,46 @@
+from json import loads
+from re import search
+from urllib.parse import unquote
+
+from scrapy.spiders import SitemapSpider
+
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+from locations.structured_data_spider import StructuredDataSpider
+
+
+class BeallsFloridaUSSpider(SitemapSpider, StructuredDataSpider):
+    name = "bealls_florida_us"
+    item_attributes = {"brand": "Bealls", "brand_wikidata": "Q4876153"}
+    allowed_domains = ["stores.beallsflorida.com"]
+    sitemap_urls = ["https://stores.beallsflorida.com/sitemap/sitemap_index.xml"]
+    sitemap_rules = [(r"stores\.beallsflorida\.com/fl/\w+/$", "parse_sd")]
+
+    def post_process_item(self, item, response, ld_data, **kwargs):
+        email = unquote(item.pop("email")).strip()
+        ref = item.pop("ref")
+
+        if m := search(
+            r"map_list_data\s*=\s*(.*)", response.xpath('//script[@type="text/javascript"]/text()').extract_first()
+        ):
+            d = loads(m.group(1))[0]
+
+            item = DictParser.parse(d)
+            item.pop("street")
+            item.pop("housenumber")
+            item["name"] = d.get("location_name")
+            item["phone"] = d.get("local_phone")
+            item["street_address"] = d.get("address_1") + d.get("address_2")
+            item["opening_hours"] = OpeningHours()
+
+            hours = loads(d.get("hours_sets:primary")).get("days")
+            for day in hours.keys():
+                item["opening_hours"].add_range(
+                    day,
+                    hours[day][0]["open"],
+                    hours[day][0]["close"],
+                )
+        item["email"] = email
+        item["ref"] = ref
+
+        yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Yettel Hungary
### Brand name

Yettel

### Wikidata ID

Q268578

### Store finder url(s)

https://www.yettel.hu/elerhetoseg/uzletkereso

I assume that it should be similar to `yettel_bg.py` and `yettel_rs.py`.


Discussion:

Resolving patch:
diff --git a/locations/spiders/yettel_hu.py b/locations/spiders/yettel_hu.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/yettel_hu.py
@@ -0,0 +1,27 @@
+from html import unescape
+
+from chompjs import parse_js_object
+from scrapy import Spider
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_HU, OpeningHours
+
+
+class YettelHUSpider(Spider):
+    name = "yettel_hu"
+    item_attributes = {"brand": "Yettel", "brand_wikidata": "Q268578"}
+    allowed_domains = ["www.yettel.hu"]
+    start_urls = ["https://www.yettel.hu/elerhetoseg/uzletkereso"]
+
+    def parse(self, response):
+        locations_js = unescape(response.xpath('//div[@id="outlet-map"]/@data-markers').get())
+        for location in parse_js_object(locations_js):
+            item = DictParser.parse(location)
+            item["ref"] = location["code"]
+            item["street_address"] = item.pop("addr_full", None)
+            hours_string = " ".join(
+                [f"{day_range}: {hours_range}" for day_range, hours_range in location["opening"].items()]
+            )
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(hours_string, days=DAYS_HU)
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Odido (PL)
### Brand name

Odido

### Wikidata ID

Q106947294

### Store finder url(s)

https://www.sklepy-odido.pl/znajdz-sklep


Discussion:
Single API call to the following URL returns all location data as a JSON object:

`https://www.sklepy-odido.pl/api/shops?itemsPerPage=10000`
Note there are no unique references I can see for each store, but the API returns an index for each store that could possibly be used (unsure of whether these indexes are stable for each store).
Currently 1664 features are available to extract.

Resolving patch:
diff --git a/locations/spiders/odido_pl.py b/locations/spiders/odido_pl.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/odido_pl.py
@@ -0,0 +1,21 @@
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+
+
+class OdidoPLSpider(Spider):
+    name = "odido_pl"
+    item_attributes = {"brand": "Odido", "brand_wikidata": "Q106947294"}
+
+    def start_requests(self):
+        yield JsonRequest(url="https://www.sklepy-odido.pl/api/shops?itemsPerPage=10000")
+
+    def parse(self, response, **kwargs):
+        for feature in response.json()["data"]:
+            item = DictParser.parse(feature)
+            item["postcode"] = item["postcode"].strip()
+            if "building" in feature:
+                item["housenumber"] = feature["building"]
+            item["ref"] = feature["index"]  # Unknown whether it's stable
+            yield item


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Good Lood (small scale chain, no Wikidata entry)
### Brand name

Good Lood

### Wikidata ID

does not have one

### Store finder url(s)

https://goodlood.com/en

I looked at https://github.com/alltheplaces/alltheplaces/blob/master/docs/PULL_REQUEST.md  and CONTRIBUTING.MD does not exists, and this one is kind of probe: are you accepting also small chains?

Or would I need to anyway run a separate project to support small chains or individual non-chain shop (in such cases monitoring and parsing opening hours data is still helpful)


Discussion:
There are spiders for small chains. I think it's a bit of a waste of time, but I still write some.

[Data](https://mobile.goodlood.com/loodspots) looks quite accessible so it shouldn't be too much trouble.

Do you want to have a go for this one?
> Do you want to have a go for this one?

I can do this, but no promises about doing it soon (my next planned step is to try to use some `opening_hours` ATP data)

Resolving patch:
diff --git a/locations/spiders/good_lood_pl.py b/locations/spiders/good_lood_pl.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/good_lood_pl.py
@@ -0,0 +1,22 @@
+from scrapy import Spider
+
+from locations.categories import Extras, apply_yes_no
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class GoodLoodPLSpider(Spider):
+    name = "good_lood_pl"
+    item_attributes = {"brand": "Good Lood", "brand_wikidata": "Q122978001"}
+    start_urls = ["https://mobile.goodlood.com/loodspots"]
+
+    def parse(self, response, **kwargs):
+        for feature in response.json():
+            if not feature["visible"]:
+                continue
+            item = DictParser.parse(feature)
+            item["image"] = feature["image"]
+            item["opening_hours"] = OpeningHours()
+            item["opening_hours"].add_ranges_from_string(feature["openingHours"])
+            apply_yes_no(Extras.WHEELCHAIR, item, feature["facilities"]["ramp"])
+            yield item


Test patch: