Repository: alltheplaces/alltheplaces

Issue:
Spider advanceautoparts is broken
During the global build at 2021-05-21-20-28-08, spider **advanceautoparts** failed with **0 features** and **405 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/logs/advanceautoparts.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/output/advanceautoparts.geojson))


Discussion:
During the global build at 2021-05-21-20-39-25, spider **advanceautoparts** failed with **0 features** and **364 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/logs/advanceautoparts.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/advanceautoparts.geojson))
During the global build at 2021-05-24-14-04-37, spider **advanceautoparts** failed with **0 features** and **377 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/logs/advanceautoparts.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/advanceautoparts.geojson))
During the global build at 2021-05-26-14-42-23, spider **advanceautoparts** failed with **0 features** and **4286 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/advanceautoparts.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/advanceautoparts.geojson))
During the global build at 2021-06-02-14-42-40, spider **advanceautoparts** failed with **0 features** and **4284 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/advanceautoparts.geojson))
During the global build at 2021-06-09-14-42-21, spider **advanceautoparts** failed with **0 features** and **4284 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/advanceautoparts.geojson))
During the global build at 2021-06-16-14-42-20, spider **advanceautoparts** failed with **0 features** and **4283 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/advanceautoparts.geojson))
During the global build at 2021-06-23-14-42-18, spider **advanceautoparts** failed with **0 features** and **4282 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/advanceautoparts.geojson))
During the global build at 2021-06-30-14-42-26, spider **advanceautoparts** failed with **0 features** and **4283 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/advanceautoparts.geojson))
During the global build at 2021-07-07-14-42-19, spider **advanceautoparts** failed with **0 features** and **4283 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/advanceautoparts.geojson))
During the global build at 2021-07-14-14-42-22, spider **advanceautoparts** failed with **0 features** and **4285 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/advanceautoparts.geojson))
During the global build at 2021-07-21-14-42-39, spider **advanceautoparts** failed with **0 features** and **4285 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/advanceautoparts.geojson))
During the global build at 2021-07-28-14-42-16, spider **advanceautoparts** failed with **0 features** and **4285 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/advanceautoparts.geojson))
During the global build at 2021-08-04-14-42-45, spider **advanceautoparts** failed with **0 features** and **4286 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/logs/advanceautoparts.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/advanceautoparts.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/advanceautoparts.geojson))

Resolving patch:
diff --git a/locations/spiders/advanceautoparts.py b/locations/spiders/advanceautoparts.py
--- a/locations/spiders/advanceautoparts.py
+++ b/locations/spiders/advanceautoparts.py
@@ -1,4 +1,6 @@
 import json
+import re
+
 import scrapy
 
 from locations.hours import OpeningHours
@@ -8,62 +10,75 @@
 class AdvanceautopartsSpider(scrapy.Spider):
 
     name = "advanceautoparts"
-    item_attributes = { 'brand': "Advance Auto Parts", 'brand_wikidata': "Q4686051" }
+    item_attributes = {"brand": "Advance Auto Parts", "brand_wikidata": "Q4686051"}
     allowed_domains = ["stores.advanceautoparts.com"]
-    download_delay = 0.1
-    start_urls = (
-        'https://stores.advanceautoparts.com/index.html#DirectoryList',
-    )
+    start_urls = ("https://stores.advanceautoparts.com/sitemap.xml",)
+
+    def parse(self, response):
+        response.selector.remove_namespaces()
+        urls = response.xpath("//loc/text()").getall()
+        storeRe = re.compile(r"^https://stores.advanceautoparts.com/[^/]+/[^/]+/[^/]+$")
+        for url in urls:
+            if storeRe.fullmatch(url):
+                yield scrapy.Request(url, callback=self.parse_store)
 
     def parse_hours(self, store_hours):
         opening_hours = OpeningHours()
 
         for weekday in store_hours:
-            day = weekday.get('day').title()
+            day = weekday.get("day").title()
             for interval in weekday.get("intervals", []):
                 open_time = str(interval.get("start"))
                 close_time = str(interval.get("end"))
-                opening_hours.add_range(day=day[:2],
-                                        open_time=open_time,
-                                        close_time=close_time,
-                                        time_format='%H%M')
+                opening_hours.add_range(
+                    day=day[:2],
+                    open_time=open_time,
+                    close_time=close_time,
+                    time_format="%H%M",
+                )
 
         return opening_hours.as_opening_hours()
 
     def parse_store(self, response):
-        brand = response.xpath('//span[@class="LocationName-brand LocationName--Advance"]/span/text()').extract_first()
-        ref = brand.replace("#","").strip()
+        name = response.xpath('//h1[@itemprop="name"]/text()').extract_first()
 
-        hours = response.xpath('//div[@class="c-hours-details-wrapper js-hours-table"]/@data-days').extract_first()
+        js = json.loads(response.xpath('//script[@class="js-map-config"]/text()').get())
+        ref = js["entities"][0]["profile"]["meta"]["id"]
+
+        hours = response.xpath(
+            '//div[@class="c-hours-details-wrapper js-hours-table"]/@data-days'
+        ).extract_first()
         try:
             opening_hours = self.parse_hours(json.loads(hours))
         except ValueError:
             opening_hours = None
 
         properties = {
-            'addr_full': response.xpath('normalize-space(//meta[@itemprop="streetAddress"]/@content)').extract_first(),
-            'phone': response.xpath('normalize-space(//div[@itemprop="telephone"]/text())').extract_first(),
-            'city': response.xpath('normalize-space(//meta[@itemprop="addressLocality"]/@content)').extract_first(),
-            'state': response.xpath('normalize-space(//abbr[@itemprop="addressRegion"]/text())').extract_first(),
-            'postcode': response.xpath('normalize-space(//span[@itemprop="postalCode"]/text())').extract_first(),
-            'ref': ref,
-            'website': response.url,
-            'lat': response.xpath('normalize-space(//meta[@itemprop="latitude"]/@content)').extract_first(),
-            'lon': response.xpath('normalize-space(//meta[@itemprop="longitude"]/@content)').extract_first(),
-            'name': response.xpath('//span[@class="LocationName-geo LocationName--Advance"]/text()').extract_first(),
-            'opening_hours': opening_hours,
-            'extras': {
-                'shop': 'car_parts',
-            },
+            "addr_full": response.xpath(
+                'normalize-space(//meta[@itemprop="streetAddress"]/@content)'
+            ).extract_first(),
+            "phone": response.xpath(
+                'normalize-space(//div[@itemprop="telephone"]/text())'
+            ).extract_first(),
+            "city": response.xpath(
+                'normalize-space(//meta[@itemprop="addressLocality"]/@content)'
+            ).extract_first(),
+            "state": response.xpath(
+                'normalize-space(//abbr[@itemprop="addressRegion"]/text())'
+            ).extract_first(),
+            "postcode": response.xpath(
+                'normalize-space(//span[@itemprop="postalCode"]/text())'
+            ).extract_first(),
+            "ref": ref,
+            "website": response.url,
+            "lat": response.xpath(
+                'normalize-space(//meta[@itemprop="latitude"]/@content)'
+            ).extract_first(),
+            "lon": response.xpath(
+                'normalize-space(//meta[@itemprop="longitude"]/@content)'
+            ).extract_first(),
+            "name": name,
+            "opening_hours": opening_hours,
+            "extras": {"shop": "car_parts"},
         }
         yield GeojsonPointItem(**properties)
-
-    def parse(self, response):
-        urls = response.xpath('//a[@class="Directory-listLink"]/@href').extract()
-        urls.extend(response.xpath('//a[@class="Teaser-titleLink"]/@href').extract())
-        for path in urls:
-            if len(path.split('/')) > 2:
-                # If there's only one store, the URL will be longer than <state code>.html
-                yield scrapy.Request(response.urljoin(path), callback=self.parse_store)
-            else:
-                yield scrapy.Request(response.urljoin(path))


Test patch: