internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: alltheplaces/alltheplaces

Issue:
Spider lees_famous_recipe is broken
During the global build at 2021-10-20-14-42-48, spider **lees_famous_recipe** failed with **0 features** and **130 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/logs/lees_famous_recipe.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/output/lees_famous_recipe.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/output/lees_famous_recipe.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/lees_famous_recipe.py b/locations/spiders/lees_famous_recipe.py
--- a/locations/spiders/lees_famous_recipe.py
+++ b/locations/spiders/lees_famous_recipe.py
@@ -83,7 +83,7 @@ def store_hours(self, hours):
                 closeH = str(int(closeH) + 12)
                 closeHours = closeH + ":" + closeM
             return dayOutput +' '+ openHours.replace(' ','') + "-" + closeHours + ';'
-        except KeyError:
+        except (KeyError, IndexError):
             return ""
 
     def parse(self, response):
@@ -99,9 +99,8 @@ def parse(self, response):
             nameString = nameString + " " + shortString
             nameString = nameString.strip()
 
-            scriptBody = response.xpath("//script[@type='text/javascript' and contains(.,'latitude')]/text()").extract_first()
-            latString = re.findall("latitude\":\"(.*?)\"", scriptBody)[0]
-            lonString = re.findall("longitude\":\"(.*?)\"", scriptBody)[0]
+            googleMapSrc = response.xpath("//*[@id='block-system-main']/div/div/iframe").extract_first()
+            [latString, lonString] = re.findall("center=(.*?)\"", googleMapSrc)[0].split(',')
 
             openingHoursString = ""
             firstHourBlock = response.xpath("//div[contains(@class,'field-name-field-hours-summer')]/div/div/p/br/parent::p/text()")
@@ -116,7 +115,7 @@ def parse(self, response):
             else:
                 countryString = "US"
                 mapUrl = response.xpath("//div[contains(@class,'map-link')]/div/a/@href").extract_first()
-                stateString = re.findall(r'(?<=\+)(.*?)(?=\+)', mapUrl)[len(re.findall(r'(?<=\+)(.*?)(?=\+)', mapUrl)) - 2].strip().replace('%2C','')
+                stateString = response.xpath("//div[contains(@class,'adr')]/div[2]/span[2]/text()").extract_first()
 
             yield GeojsonPointItem(
                 ref=nameString,
@@ -125,7 +124,7 @@ def parse(self, response):
                 opening_hours=openingHoursString,
                 state=stateString,
                 postcode=response.xpath("//div[@class='city-state-zip']/span[@class='postal-code']/text()").extract_first().strip(),
-                phone=self.parse_phone(response.xpath("//div[contains(@class,'field-name-field-phone')]/div/div/text()").extract_first().strip()),
+                phone=self.parse_phone(response.xpath("//div[contains(@class,'adr')]/div[3]/text()").extract_first().strip()),
                 country = countryString,
                 lat=float(latString),
                 lon=float(lonString),


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider ingles is broken
During the global build at 2021-07-07-14-42-19, spider **ingles** failed with **0 features** and **189 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/logs/ingles.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/ingles.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/ingles.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/ingles.py b/locations/spiders/ingles.py
--- a/locations/spiders/ingles.py
+++ b/locations/spiders/ingles.py
@@ -37,14 +37,14 @@ def parse_store(self, response):
             'addr_full': response.meta["addr_full"],
             'city': response.meta["city"],
             'state': response.meta["state"],
-            'postcode': re.search(r'(\d{5})',response.xpath("/html/body/fieldset/div[2]/span[2]/strong/text()").get()).group(),
+            'postcode': re.search(r'(\d{5})',response.xpath("/html/body/div[2]/span[2]/strong/text()").get()).group(),
             'phone': response.xpath("/html/body/fieldset/div[2]/a/text()").get(),
             'lat': response.meta["lat"],
             'lon': response.meta["lon"],
             'website': response.url,
         }
 
-        hours = self.parse_hours(" ".join(response.xpath("/html/body/fieldset/div[2]/text()")[2].getall()).strip())
+        hours = self.parse_hours(" ".join(response.xpath("/html/body/fieldset/div[2]/text()")[1].getall()).strip())
         if hours:
             properties["opening_hours"] = hours
 


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider konagrill is broken
During the global build at 2021-06-16-14-42-20, spider **konagrill** failed with **0 features** and **24 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/logs/konagrill.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/konagrill.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/konagrill.geojson))
Spider konagrill is broken
During the global build at 2021-10-13-14-42-23, spider **konagrill** failed with **0 features** and **24 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/logs/konagrill.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/output/konagrill.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/output/konagrill.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/kona_grill.py b/locations/spiders/kona_grill.py
--- a/locations/spiders/kona_grill.py
+++ b/locations/spiders/kona_grill.py
@@ -88,6 +88,10 @@ def parse_hours(self, hours):
         oh = OpeningHours()
 
         for t in hours:
+            # Some day entries contain invalid week data, e.g. "Brunch"
+            # "Brunch" is a special dining hour that is contained in regular hours, ignore it
+            if "Brunch" in t.get("days"):
+                continue
             days = self.parse_days(t.get("days"))
             open_time, close_time = t.get("hours").split("-")
             ot = open_time.strip()


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider subway is broken
During the global build at 2021-09-15-14-42-44, spider **subway** failed with **31396 features** and **22 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/logs/subway.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/subway.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/subway.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/subway.py b/locations/spiders/subway.py
--- a/locations/spiders/subway.py
+++ b/locations/spiders/subway.py
@@ -39,7 +39,9 @@ def parse(self, response):
     def parse_restaurant(self, js):
         # Note: Of the five different coordinate fields, this is the one that always exists
         lat_long = js["profile"]["yextDisplayCoordinate"]
-        website = urlparse(js["profile"]["websiteUrl"])._replace(query="").geturl()
+        website = ""
+        if 'websiteUrl' in js["profile"]:
+            website = urlparse(js["profile"]["websiteUrl"])._replace(query="").geturl()
         properties = {
             "lat": lat_long["lat"],
             "lon": lat_long["long"],


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider avis is broken
During the global build at 2021-06-02-14-42-40, spider **avis** failed with **4383 features** and **36 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/logs/avis.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/avis.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/avis.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/avis.py b/locations/spiders/avis.py
--- a/locations/spiders/avis.py
+++ b/locations/spiders/avis.py
@@ -66,6 +66,15 @@ def clean(val):
 
         ref = response.url.split('/')[-1]
 
+        latitude = None
+        longitude = None
+
+        if response.xpath('//meta[@itemprop="latitude"]/@content').extract_first() is not None:
+            latitude = float(response.xpath('//meta[@itemprop="latitude"]/@content').extract_first())
+
+        if response.xpath('//meta[@itemprop="longitude"]/@content').extract_first() is not None:
+            longitude = float(response.xpath('//meta[@itemprop="longitude"]/@content').extract_first())
+
         properties = {
             'name': clean(response.xpath('//h2/span[@itemprop="name"]/text()').extract_first()),
             'addr_full': clean(response.xpath('normalize-space(//span[@itemprop="streetAddress"]/text())').extract_first()),
@@ -76,8 +85,8 @@ def clean(val):
             'country': clean(response.xpath('normalize-space(//span[@itemprop="addressCountry"]/text())').extract_first()),
             'ref': ref,
             'website': response.url,
-            'lat':  float(response.xpath('//meta[@itemprop="latitude"]/@content').extract_first()),
-            'lon':  float(response.xpath('//meta[@itemprop="longitude"]/@content').extract_first()),
+            'lat':  latitude,
+            'lon':  longitude,
         }
         hours = response.xpath('//meta[@itemprop="openingHours"]/@content').extract_first()
         if hours:


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider anytime_fitness is broken
During the global build at 2021-09-29-14-42-48, spider **anytime_fitness** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/logs/anytime_fitness.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/output/anytime_fitness.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/output/anytime_fitness.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/anytime_fitness.py b/locations/spiders/anytime_fitness.py
--- a/locations/spiders/anytime_fitness.py
+++ b/locations/spiders/anytime_fitness.py
@@ -7,27 +7,27 @@
 
 class AnytimeFitnessSpider(scrapy.Spider):
     name = 'anytime_fitness'
-    item_attributes = { 'brand': "Anytime Fitness" }
+    item_attributes = {'brand': "Anytime Fitness", 'brand_wikidata': "Q4778364"}
     allowed_domains = ['www.anytimefitness.com']
 
     def start_requests(self):
-        url = 'https://www.anytimefitness.com/wp-content/uploads/gyms.json'
+        url = 'https://www.anytimefitness.com/wp-content/uploads/locations.json'
         yield scrapy.Request(url, callback=self.parse)
 
     def parse(self, response):
-        gyms = json.loads(response.body_as_unicode())
+        gyms = json.loads(response.text)
 
         for gym in gyms:
             yield GeojsonPointItem(
-                lat = gym['latitude'],
-                lon = gym['longitude'],
-                addr_full = ", ".join(filter(None, [gym['content']['address'], gym['content']['address2']])),
-                city = gym['content']['city'],
-                phone = gym['content']['phone'],
-                state = gym['content']['state_abbr'],
-                postcode = gym['content']['zip'],
-                ref = gym['content']['url'],
-                country = gym['content']['country'],
-                name = html.unescape(gym['content']['title']),
-                extras = {"number": gym['content']['number']}
-                )
+                lat=gym['latitude'],
+                lon=gym['longitude'],
+                addr_full=", ".join(filter(None, [gym['content']['address'], gym['content']['address2']])),
+                city=gym['content']['city'],
+                phone=gym['content']['phone'],
+                state=gym['content']['state_abbr'],
+                postcode=gym['content']['zip'],
+                ref=gym['content']['url'],
+                country=gym['content']['country'],
+                name=html.unescape(gym['content']['title']),
+                extras={"number": gym['content']['number']}
+            )


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
BMO Harris Bank
https://branchlocator.bmoharris.com/


Discussion:
(General scraper added, but could still use opening hours included)

Resolving patch:
diff --git a/locations/spiders/bmo_harris.py b/locations/spiders/bmo_harris.py
--- a/locations/spiders/bmo_harris.py
+++ b/locations/spiders/bmo_harris.py
@@ -7,13 +7,14 @@
 
 
 class BMOHarrisSpider(scrapy.Spider):
-    name = "bmo-harris"
-    item_attributes = { 'brand': "BMO Harris Bank" }
+    name = "bmo_harris"
+    item_attributes = {'brand': "BMO Harris Bank", 'brand_wikidata': "Q4835981"}
     allowed_domains = ["branches.bmoharris.com"]
     download_delay = 0.5
     start_urls = (
         'https://branches.bmoharris.com/',
     )
+    user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
 
     def parse_store(self, response):
         properties = {
@@ -33,9 +34,9 @@ def parse_store(self, response):
 
     def parse(self, response):
         # Step into hierarchy of place
-        for url in response.xpath("//div[@class='itemlist']/p/a/@href").extract():
+        for url in response.xpath("//ul[@class='itemlist']/li/a/@href").extract():
             yield scrapy.Request(response.urljoin(url))
 
         # Look for links to stores
-        for url in response.xpath("//div[@class='itemlist']/li/span[@itemprop='streetAddress']/a/@href").extract():
+        for url in response.xpath("//ul[@class='itemlist']/li/div/span[@itemprop='streetAddress']/a/@href").extract():
             yield scrapy.Request(response.urljoin(url), callback=self.parse_store)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Cornerstone Healthcare Group
https://www.chghospitals.com/locations/
https://www.chghospitals.com/wp-json/wp/v2/location?per_page=100


Discussion:

Resolving patch:
diff --git a/locations/spiders/cornerstone_healthcare_group.py b/locations/spiders/cornerstone_healthcare_group.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/cornerstone_healthcare_group.py
@@ -0,0 +1,33 @@
+import json
+
+import scrapy
+
+from locations.items import GeojsonPointItem
+
+
+class Cornerstone_healthcare_groupSpider(scrapy.Spider):
+    name = "cornerstone_healthcare_group"
+    item_attributes = {'brand': 'Cornerstone Healthcare Group'}
+    allowed_domains = ['chghospitals.com']
+    start_urls = [
+        'https://www.chghospitals.com/wp-json/wp/v2/location?per_page=100',
+    ]
+
+    def parse(self, response):
+        data = json.loads(response.body_as_unicode())
+
+        for i, item in enumerate(data):
+            properties = {
+                'name': item['title']['rendered'],
+                'addr_full': item['acf']['address'],
+                'city': item['acf']['city'],
+                'state': item['acf']['state'],
+                'postcode': item['acf']['zip'],
+                'country': 'US',
+                'ref': item['link'],
+                'website': item['link'],
+                'phone': item['acf']['phone'],
+                'lat': item['acf']['latitude'],
+                'lon': item['acf']['longitude'],
+            }
+            yield GeojsonPointItem(**properties)
\ No newline at end of file


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Verve Coffee Roasters
https://www.vervecoffee.com/pages/locations-all


Discussion:

Resolving patch:
diff --git a/locations/spiders/verve_coffee_roasters.py b/locations/spiders/verve_coffee_roasters.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/verve_coffee_roasters.py
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+import scrapy
+
+from locations.items import GeojsonPointItem
+from locations.hours import OpeningHours
+
+
+class VerveCoffeeRoastersSpider(scrapy.Spider):
+    name = "verve_coffee_roasters"
+    item_attributes = {'brand': 'Verve Coffee Roasters'}
+    allowed_domains = ['www.vervecoffee.com/']
+    start_urls = [
+        'https://www.vervecoffee.com/pages/locations-all',
+    ]
+
+    def parse_hours(self, hours):
+        opening_hours = OpeningHours()
+
+        days, store_hours = hours.split()
+        open_time, close_time = store_hours.split('-')
+        for day in days.split(','):
+            opening_hours.add_range(day=day, open_time=open_time, close_time=close_time, time_format='%H:%M')
+
+        return opening_hours.as_opening_hours()
+
+
+    def parse(self, response):
+        stores = response.xpath('//section[@itemtype="http://schema.org/CafeOrCoffeeShop"]')
+        
+        for store in stores:
+            properties = {
+                'ref': store.xpath('.//div[@class="info text-align--center"]/a/text()').extract_first().strip(),
+                'name': store.xpath('.//div[@class="info text-align--center"]/a/text()').extract_first().strip(),
+                'addr_full': store.xpath('normalize-space(.//span[@itemprop="streetAddress"]//text())').extract_first(),
+                'city': store.xpath('normalize-space(.//span[@itemprop="addressLocality"]//text())').extract_first(),
+                'state': store.xpath('normalize-space(.//span[@itemprop="addressRegion"]//text())').extract_first(),
+                'postcode': store.xpath('normalize-space(.//span[@itemprop="postalCode"]//text())').extract_first(),
+                'phone': store.xpath('normalize-space(.//a[@itemprop="telephone"]//text())').extract_first(),
+            }
+            
+            hours = store.xpath('.//meta[@itemprop="openingHours"]/@content').extract_first()
+            if hours:
+                properties['opening_hours'] = self.parse_hours(hours)
+
+            yield GeojsonPointItem(**properties)


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Spider public_storage is broken
During the global build at 2021-08-04-14-42-45, spider **public_storage** failed with **834 features** and **1879 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/logs/public_storage.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/public_storage.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/public_storage.geojson))


Discussion:

Resolving patch:
diff --git a/locations/spiders/public_storage.py b/locations/spiders/public_storage.py
--- a/locations/spiders/public_storage.py
+++ b/locations/spiders/public_storage.py
@@ -20,9 +20,13 @@ def parse(self, response):
         for path in city_urls:
             yield scrapy.Request(
                 path.strip(),
-                callback=self.parse_store,
+                callback=self.load_store,
             )
 
+    def load_store(self, response):
+        ldjson = response.xpath('//link[@type="application/ld+json"]/@href').get()
+        yield scrapy.Request(response.urljoin(ldjson), callback=self.parse_store)
+
     def parse_hours(self, hours):
         opening_hours = OpeningHours()
 
@@ -37,11 +41,11 @@ def parse_hours(self, hours):
         return opening_hours.as_opening_hours()
 
     def parse_store(self, response):
-        data = json.loads(response.xpath('//script[@type="application/ld+json"]/text()').extract_first())
-        data = data['@graph'][0]
+        data = response.json()['@graph'][0]
 
         properties = {
             "ref": data['@id'],
+            "website": data['url'],
             "opening_hours": self.parse_hours(data['openingHoursSpecification']),
             "addr_full": data['address']['streetAddress'],
             "city": data['address']['addressLocality'],


Test patch:
Repository: alltheplaces/alltheplaces

Issue:
Coles



Discussion:

Resolving patch:
diff --git a/locations/spiders/coles_au.py b/locations/spiders/coles_au.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/coles_au.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+import re
+
+import scrapy
+import json
+
+from locations.items import GeojsonPointItem
+
+class ColesSpider(scrapy.Spider):
+    name = "coles_au"
+    item_attributes = {'brand': "Coles"}
+    allowed_domains = ["apigw.coles.com.au"]
+    start_urls = [
+        'https://apigw.coles.com.au/digital/colesweb/v1/stores/search?latitude=-28.014547&longitude=135.171168&brandIds=2,1&numberOfStores=10',
+    ]
+
+    def parse(self, response):
+        with open('./locations/searchable_points/au_centroids_20km_radius.csv') as points:
+            next(points)
+            for point in points:
+                row = point.replace('\n', '').split(',')
+                lati = row[1]
+                long = row[2]
+                searchurl = 'https://apigw.coles.com.au/digital/colesweb/v1/stores/search?latitude={la}&longitude={lo}&brandIds=2,1&numberOfStores=10'.format(la=lati, lo=long)
+                yield scrapy.Request(response.urljoin(searchurl), callback=self.parse_search)
+
+    def parse_search(self, response):
+        data = json.loads(json.dumps(response.json()))
+
+        for i in data['stores']:
+            properties = {
+                'ref': i['storeId'],
+                'name': i['brandName'],
+                'addr_full': i['address'],
+                'city': i['suburb'],
+                'state': i['state'],
+                'postcode': i['postcode'],
+                'country': "AU",
+                'phone': i['phone'],
+                'lat': i['latitude'],
+                'lon': i['longitude'],
+            }
+
+            yield GeojsonPointItem(**properties)


Test patch: