internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
Spider cvs is broken
During the global build at 2021-08-18-14-42-26, spider **cvs** failed with **0 features** and **9870 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/logs/cvs.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/cvs.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/cvs.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/cvs.py b/locations/spiders/cvs.py
--- a/locations/spiders/cvs.py
+++ b/locations/spiders/cvs.py
@@ -77,8 +77,8 @@ def parse_stores(self, response):
             'country': data["address"]["addressCountry"],
             'phone': data["address"].get("telephone"),
             'website': data.get("url") or response.url,
-            'lat': float(data["geo"]["latitude"]),
-            'lon': float(data["geo"]["longitude"]),
+            'lat': data["geo"]["latitude"] or None,
+            'lon': data["geo"]["longitude"] or None,
         }
 
         hours = self.parse_hours(data["openingHours"])


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider costco is broken
During the global build at 2021-08-18-14-42-26, spider **costco** failed with **0 features** and **2 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/logs/costco.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/costco.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/costco.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/costco.py b/locations/spiders/costco.py
--- a/locations/spiders/costco.py
+++ b/locations/spiders/costco.py
@@ -28,7 +28,7 @@ class CostcoSpider(scrapy.Spider):
         'https://www.costco.com/warehouse-locations',
     )
     custom_settings = {
-        'USER_AGENT': 'Mozilla/5.0',
+        'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
     }
 
     download_delay = 0.5


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider sheetz is broken
During the global build at 2021-08-11-14-42-19, spider **sheetz** failed with **526 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/logs/sheetz.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/sheetz.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/sheetz.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/sheetz.py b/locations/spiders/sheetz.py
--- a/locations/spiders/sheetz.py
+++ b/locations/spiders/sheetz.py
@@ -24,8 +24,8 @@ def parse(self, response):
                 'ref': store['storeNumber'],
                 'phone': store.get('phone'),
                 'website': 'https://orderz.sheetz.com/#/main/location/store/'+store['storeNumber'],
-                'lat': float(store['latitude']),
-                'lon': float(store['longitude']),
+                'lat': store['latitude'],
+                'lon': store['longitude'],
                 'opening_hours': '24/7' if store['open24x7'] else None,
                 'extras': {
                     'amenity:chargingstation': store['evCharger'],


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider texas_roadhouse is broken
During the global build at 2021-08-11-14-42-19, spider **texas_roadhouse** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/logs/texas_roadhouse.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/texas_roadhouse.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/texas_roadhouse.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/texas_roadhouse.py b/locations/spiders/texas_roadhouse.py
--- a/locations/spiders/texas_roadhouse.py
+++ b/locations/spiders/texas_roadhouse.py
@@ -6,7 +6,10 @@
 
 class TexasRoadhouseSpider(scrapy.Spider):
     name = "texas_roadhouse"
-    item_attributes = { 'brand': "Texas Roadhouse", 'brand_wikidata': "Q7707945" }
+    item_attributes = {
+        'brand': "Texas Roadhouse",
+        'brand_wikidata': "Q7707945",
+    }
     allowed_domains = ["www.texasroadhouse.com"]
     start_urls = (
         'https://www.texasroadhouse.com/sitemap.xml',


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Leslie's Poolmart
https://lesliespool.com/stores


Discussion:

Resolving patch:
diff --git a/locations/spiders/leslies_poolmart.py b/locations/spiders/leslies_poolmart.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/leslies_poolmart.py
@@ -0,0 +1,53 @@
+# -*- coding: utf-8 -*-
+import re
+
+import scrapy
+
+from locations.items import GeojsonPointItem
+
+
+class LesliesPoolmartSpider(scrapy.Spider):
+    name = "leslies_poolmart"
+    item_attributes = {'brand': "Leslie's Poolmart"}
+    allowed_domains = ["lesliespool.com"]
+    start_urls = [
+        'https://lesliespool.com/sitemap_2.xml',
+    ]
+    download_delay = 0.3
+
+    def parse(self, response):
+        response.selector.remove_namespaces()
+        urls = response.xpath('//url/loc[contains(text(), "/location/")]/text()').extract()
+
+        for url in urls:
+            yield scrapy.Request(url=url, callback=self.parse_store)
+
+    def parse_store(self, response):
+        store_title = response.xpath('//h1/text()').extract_first()
+
+        # Handle handful of empty pages
+        if store_title:
+            ref = re.search(r'.*\s#?([0-9]*)', store_title).group(1)
+
+            city_state_postal = response.xpath('//h5[@class="store-detail-address"]/span[3]/text()').extract_first()
+
+            if city_state_postal:  # Get store details, otherwise pass if empty page
+                city, state, zipcode = re.search(r'^(.*),\s*([a-z]{2})\s*([0-9]{5}).*$',
+                                                 city_state_postal.strip(),
+                                                 re.IGNORECASE).groups()
+
+                properties = {
+                    'ref': ref,
+                    'name': response.xpath('//h5[@class="store-detail-address"]/span[1]/text()').extract_first(),
+                    'addr_full': response.xpath('//h5[@class="store-detail-address"]/span[2]/text()').extract_first(),
+                    'city': city,
+                    'state': state,
+                    'postcode': zipcode,
+                    'country': 'US',
+                    'lat': float(response.xpath('//a[contains(@class, "select-store-button")]/@data-store-latitude').extract_first()),
+                    'lon': float(response.xpath('//a[contains(@class, "select-store-button")]/@data-store-longitude').extract_first()),
+                    'phone': response.xpath('//h5[contains(@class, "store-detail-phone")]/text()').extract_first(),
+                    'website': response.url
+                }
+
+                yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Acme Markets
https://local.acmemarkets.com/index.html


Discussion:

Resolving patch:
diff --git a/locations/spiders/acme_markets.py b/locations/spiders/acme_markets.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/acme_markets.py
@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+import re
+
+import scrapy
+
+from locations.items import GeojsonPointItem
+
+
+class AcmeMarketsSpider(scrapy.Spider):
+    name = "acme_markets"
+    item_attributes = {'brand': 'Acme Markets', 'brand_wikidata': 'Q341975'}
+    allowed_domains = ['acmemarkets.com']
+    start_urls = [
+        'https://local.acmemarkets.com/sitemap.xml',
+    ]
+    download_delay = 0.3
+
+    def parse(self, response):
+        response.selector.remove_namespaces()
+        urls = response.xpath("//loc/text()").getall()
+
+        for url in urls:
+            if url.count('/') == 5:
+                yield scrapy.Request(url, callback=self.parse_store)
+
+    def parse_store(self, response):
+        ref = re.search(r'.+/(.+?)/?(?:\.html|$)', response.url).group(1)
+
+        properties = {
+            'ref': ref,
+            'name': response.xpath('normalize-space(//div[@class="Heading--lead ContentBanner-title"]/text())').extract_first(),
+            'addr_full': response.xpath('//meta[@itemprop="streetAddress"]/@content').extract_first(),
+            'city': response.xpath('//meta[@itemprop="addressLocality"]/@content').extract_first(),
+            'state': response.xpath('normalize-space(//abbr[@itemprop="addressRegion"]//text())').extract_first(),
+            'postcode': response.xpath('normalize-space(//span[@itemprop="postalCode"]//text())').extract_first(),
+            'country': 'US',
+            'phone': response.xpath('normalize-space(//a[@class="Phone-link"]//text())').extract_first(),
+            'website': response.url,
+            'lat': float(response.xpath('normalize-space(//meta[@itemprop="latitude"]/@content)').extract_first()),
+            'lon': float(response.xpath('normalize-space(//meta[@itemprop="longitude"]/@content)').extract_first()),
+        }
+
+        yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider arby is broken
During the global build at 2021-05-26-14-42-23, spider **arby** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/arby.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/arby.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/arby.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/arbys.py b/locations/spiders/arbys.py
--- a/locations/spiders/arbys.py
+++ b/locations/spiders/arbys.py
@@ -12,24 +12,31 @@ class ArbysSpider(scrapy.Spider):
     allowed_domains = ["locations.arbys.com"]
     download_delay = 0.2
     start_urls = (
-        'https://locations.arbys.com/browse/',
+        'https://locations.arbys.com/',
     )
 
     def get_store_info(self, response):
         data = response.xpath('//script[@type="application/ld+json"]/text()').extract_first()
         if data:
-            data = json.loads(data)[0]
+            try:
+                data = json.loads(data)[0]
+            except json.JSONDecodeError:
+                # Unescaped " on two pages
+                lines = data.split('\n')
+                i = 2 + next(i for (i, line) in enumerate(lines) if "mainContentOfPage" in line)
+                lines[i] = '"text": ""}'
+                data = '\n'.join(lines)
+                data = json.loads(data)[0]
+
 
             properties = {
-                # store name is after the pipe, e.g. Fast Food Drive-Thru
-                # Restaurants | Arby's 8437
-                'ref': data["name"].rsplit("|", 1)[-1].strip(),
+                'ref': response.css('div.store-id::text').get().split(': ')[-1],
                 'name': data["name"],
                 'addr_full': data["address"]["streetAddress"].strip(),
                 'city': data["address"]["addressLocality"].strip(),
                 'state': data["address"]["addressRegion"],
                 'postcode': data["address"]["postalCode"],
-                'phone': data.get("telephone", None),
+                'phone': data["address"]["telephone"],
                 'lat': float(data["geo"]["latitude"]),
                 'lon': float(data["geo"]["longitude"]),
                 'website': response.url,


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider planned_parenthood is broken
During the global build at 2021-05-26-14-42-23, spider **planned_parenthood** failed with **598 features** and **598 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/planned_parenthood.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/planned_parenthood.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/planned_parenthood.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/planned_parenthood.py b/locations/spiders/planned_parenthood.py
--- a/locations/spiders/planned_parenthood.py
+++ b/locations/spiders/planned_parenthood.py
@@ -27,9 +27,14 @@ def parse_state(self, response):
             yield scrapy.Request(
                 response.urljoin(path),
                 callback=self.parse_venue,
+                meta={'dont_redirect':True}
             )
 
     def parse_venue(self, response):
+        if response is None:
+            # Ignoring redirects
+            return
+
         properties = {
             'addr_full': response.xpath('//*[@itemprop="streetAddress"]/text()').extract_first(),
             'city': response.xpath('//*[@itemprop="addressLocality"]/text()').extract_first(),
@@ -42,7 +47,7 @@ def parse_venue(self, response):
 
         map_image_url = response.xpath('//img[@class="address-map"]/@src').extract_first()
         match = re.search(r"center=(.*?),(.*?)&zoom", map_image_url)
-        properties['lat'] = float(match.group(1)),
-        properties['lon'] = float(match.group(2)),
+        properties['lat'] = float(match.group(1))
+        properties['lon'] = float(match.group(2))
 
         yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Palm Beach Tan
https://palmbeachtan.com/locations/states/


Discussion:

Resolving patch:
diff --git a/locations/spiders/palm_beach_tan.py b/locations/spiders/palm_beach_tan.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/palm_beach_tan.py
@@ -0,0 +1,51 @@
+# -*- coding: utf-8 -*-
+import re
+
+import scrapy
+
+from locations.items import GeojsonPointItem
+
+
+class PalmBeachTanSpider(scrapy.Spider):
+    name = "palm_beach_tan"
+    item_attributes = {'brand': 'Palm Beach Tan'}
+    allowed_domains = ['palmbeachtan.com']
+    start_urls = [
+        'https://palmbeachtan.com/locations/states/',
+    ]
+    download_delay = 0.5
+
+    def parse(self, response):
+        urls = response.xpath('//div[@class="state-list"]/ul/li/a/@href').extract()
+
+        for url in urls:
+            yield scrapy.Request(response.urljoin(url), callback=self.parse_cities)
+
+    def parse_cities(self, response):
+        city_urls = response.xpath('//ul[@id="states-list"]/li/a/@href').extract()
+
+        for city_url in city_urls:
+            yield scrapy.Request(response.urljoin(city_url), callback=self.parse_store)
+
+    def parse_store(self, response):
+        ref = re.search(r'.+/(.+?)/?(?:\.html|$)', response.url).group(1)
+
+        name = response.xpath('//div[@class="main-copy"]/h2/text()').extract_first()
+        for junk in ((" - NOW HIRING!", ""), (" - Now Hiring!", ""), (" - NOW OPEN!", ""), (" - Now Open!", "")):
+            name = name.replace(*junk)
+
+        properties = {
+            'ref': ref,
+            'name': name,
+            'addr_full': response.xpath('//div[@class="location-info"]//data[@itemprop="streetAddress"]/text()').extract_first(),
+            'city': response.xpath('//div[@class="location-info"]//data[@itemprop="addressLocality"]/text()').extract_first(),
+            'state': response.xpath('//div[@class="location-info"]//data[@itemprop="addressRegion"]/text()').extract_first(),
+            'postcode': response.xpath('//div[@class="location-info"]//data[@itemprop="postalCode"]/text()').extract_first(),
+            'country': 'US',
+            'lat': response.xpath('//span[@class="directions"]/@data-lat').extract_first(),
+            'lon': response.xpath('//span[@class="directions"]/@data-lng').extract_first(),
+            'phone': response.xpath('//div[@class="location-info"]//data[@itemprop="telephone"]/text()').extract_first(),
+            'website': response.url
+        }
+
+        yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider pricerite is broken
During the global build at 2021-05-26-14-42-23, spider **pricerite** failed with **0 features** and **2 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/pricerite.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/pricerite.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/pricerite.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/pricerite.py b/locations/spiders/pricerite.py
--- a/locations/spiders/pricerite.py
+++ b/locations/spiders/pricerite.py
@@ -5,32 +5,36 @@
 import scrapy
 
 from locations.items import GeojsonPointItem
-from locations.hours import OpeningHours
 
 
 class PriceRiteSpider(scrapy.Spider):
     name = "pricerite"
     item_attributes = { 'brand': "PriceRite" }
-    allowed_domains = ["priceritesupermarkets.com"]
+    allowed_domains = ["priceritemarketplace.com"]
 
     start_urls = (
-        "https://www.priceritesupermarkets.com/locations/",
+        "https://www.priceritemarketplace.com/",
     )
 
     def parse(self, response):
-        script = response.xpath('//script[contains(text(), "var stores")]').extract_first()
-        stores = json.loads(re.search(r'var stores = (.*?);', script).groups()[0])
+        script = response.xpath('//script[contains(text(), "__PRELOADED_STATE__")]/text()').extract_first()
+        script = script[script.index('{'):]
+        stores = json.loads(script)['stores']['availablePlanningStores']['items']
 
         for store in stores:
+            ref = store["retailerStoreId"]
             properties = {
-                "ref": store["storeNumber"],
+                "ref": ref,
+                "website": f"https://www.priceritemarketplace.com/sm/planning/rsid/{ref}",
                 "name": store["name"],
-                "lat": store["latitude"],
-                "lon": store["longitude"],
-                "addr_full": store["address1"],
+                "lat": store["location"]["latitude"],
+                "lon": store["location"]["longitude"],
+                "addr_full": store["addressLine1"],
                 "city": store["city"],
-                "state": store["state"],
-                "postcode": store["zipCode"],
+                "state": store["countyProvinceState"],
+                "postcode": store["postCode"],
+                "phone": store["phone"],
+                "opening_hours": store["openingHours"],
             }
 
             yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider regis_uk is broken
During the global build at 2021-05-26-14-42-23, spider **regis_uk** failed with **33 features** and **35 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/regis_uk.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/regis_uk.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/regis_uk.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/regis_salon_uk.py b/locations/spiders/regis_salon_uk.py
--- a/locations/spiders/regis_salon_uk.py
+++ b/locations/spiders/regis_salon_uk.py
@@ -11,6 +11,7 @@ class RegisUKSpider(scrapy.Spider):
     item_attributes = { 'brand': "Regis Salon" }
     allowed_domains = ["www.regissalons.co.uk"]
     start_urls = ['https://www.regissalons.co.uk/salon-locator?show-all=yes']
+    download_delay = 4.0
 
     def convert_hours(self, hours):
         hours = [x.strip() for x in hours]
@@ -21,20 +22,20 @@ def convert_hours(self, hours):
                 from_hr, to_hr = [hr.strip() for hr in hours[i].split('–')]
                 if re.search(regex_am, from_hr):
                     from_hr = re.sub(regex_am, '', from_hr)
-                    hour_min = from_hr.split(':')
+                    hour_min = re.split('[:.]', from_hr)
                     if len(hour_min[0]) < 2:
                         hour_min[0].zfill(2)
                     converted_times += (":".join(hour_min)) + ' - '
                 else:
                     from_hr = re.sub(regex_pm, '', from_hr)
-                    hour_min = from_hr.split(':')
+                    hour_min = re.split('[:.]', from_hr)
                     if int(hour_min[0]) < 12:
                         hour_min[0] = str(12 + int(hour_min[0]))
                     converted_times += (":".join(hour_min)) + ' - '
 
                 if re.search(regex_am, to_hr):
                     to_hr = re.sub(regex_am, '', to_hr)
-                    hour_min = to_hr.split(':')
+                    hour_min = re.split('[:.]', to_hr)
                     if len(hour_min[0]) < 2:
                         hour_min[0].zfill(2)
                     if int(hour_min[0]) == 12:
@@ -42,7 +43,7 @@ def convert_hours(self, hours):
                     converted_times += (":".join(hour_min))
                 else:
                     to_hr = re.sub(regex_pm, '', to_hr)
-                    hour_min = to_hr.split(':')
+                    hour_min = re.split('[:.]', to_hr)
                     if int(hour_min[0]) < 12:
                         hour_min[0] = str(12 + int(hour_min[0]))
                     converted_times += (":".join(hour_min))
@@ -77,4 +78,6 @@ def parse_store(self, response):
     def parse(self, response):
         stores = response.xpath('//ul[@class="list"]//a/@href').extract()
         for store in stores:
+            if '/salon-region/' in store:
+                continue
             yield scrapy.Request(store, callback=self.parse_store)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider rentacenter is broken
During the global build at 2021-05-26-14-42-23, spider **rentacenter** failed with **2196 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/rentacenter.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/rentacenter.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/rentacenter.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/rentacenter.py b/locations/spiders/rentacenter.py
--- a/locations/spiders/rentacenter.py
+++ b/locations/spiders/rentacenter.py
@@ -48,15 +48,15 @@ def parse_location(self, response):
 
         properties = {
             'addr_full': data["address"]["streetAddress"],
-            'phone': data["telephone"],
+            'phone': data.get("telephone"),
             'city': data["address"]["addressLocality"],
             'state': data["address"]["addressRegion"],
             'postcode': data["address"]["postalCode"],
             'country': 'US',
             'ref': ref,
             'website': response.url,
-            'lat': float(data["geo"]["latitude"]),
-            'lon': float(data["geo"]["longitude"]),
+            'lat': data["geo"]["latitude"],
+            'lon': data["geo"]["longitude"],
             'name': data["name"]
         }
 
@@ -87,5 +87,7 @@ def parse(self, response):
         urls = [url.strip() for url in urls]
 
         for url in urls:
+            if '/home/' in url:
+                continue
             yield scrapy.Request(url, callback=self.parse_state_sitemap)
 


Test patch: