Repository: alltheplaces/alltheplaces

Issue:
Spider ace_hardware is broken
During the global build at 2021-05-21-20-28-08, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/logs/ace_hardware.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-21-20-28-08/output/ace_hardware.geojson))


Discussion:
During the global build at 2021-05-21-20-39-25, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/logs/ace_hardware.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/ace_hardware.geojson))
During the global build at 2021-05-24-14-04-37, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/logs/ace_hardware.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/ace_hardware.geojson))
During the global build at 2021-05-26-14-42-23, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/ace_hardware.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/ace_hardware.geojson))
During the global build at 2021-06-02-14-42-40, spider **ace_hardware** failed with **25 features** and **3974 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/ace_hardware.geojson))
During the global build at 2021-06-09-14-42-21, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/ace_hardware.geojson))
During the global build at 2021-06-16-14-42-20, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/ace_hardware.geojson))
During the global build at 2021-06-23-14-42-18, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/ace_hardware.geojson))
During the global build at 2021-06-30-14-42-26, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/ace_hardware.geojson))
During the global build at 2021-07-07-14-42-19, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/ace_hardware.geojson))
During the global build at 2021-07-14-14-42-22, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/ace_hardware.geojson))
During the global build at 2021-07-21-14-42-39, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/ace_hardware.geojson))
During the global build at 2021-07-28-14-42-16, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/ace_hardware.geojson))
During the global build at 2021-08-04-14-42-45, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/ace_hardware.geojson))
During the global build at 2021-08-11-14-42-19, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/ace_hardware.geojson))
During the global build at 2021-08-18-14-42-26, spider **ace_hardware** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/logs/ace_hardware.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/ace_hardware.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/ace_hardware.geojson))

Resolving patch:
diff --git a/locations/settings.py b/locations/settings.py
--- a/locations/settings.py
+++ b/locations/settings.py
@@ -16,6 +16,8 @@
 NEWSPIDER_MODULE = 'locations.spiders'
 
 
+# Crawl responsibly by identifying yourself (and your website) on the user-agent
+USER_AGENT = 'Mozilla/5.0 (Linux; rv:1.0) https://github.com/alltheplaces/alltheplaces'
 
 # Obey robots.txt rules
 ROBOTSTXT_OBEY = False
@@ -58,19 +60,9 @@
 
 # Enable or disable downloader middlewares
 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
-DOWNLOADER_MIDDLEWARES = {
-    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
-    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
-    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
-    'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
-}
-
-FAKEUSERAGENT_PROVIDERS = [
-    'scrapy_fake_useragent.providers.FakeUserAgentProvider',
-    'scrapy_fake_useragent.providers.FakerProvider',
-    'scrapy_fake_useragent.providers.FixedUserAgentProvider',
-]
-USER_AGENT = 'Mozilla/5.0 (Linux; rv:1.0) https://github.com/alltheplaces/alltheplaces'
+#DOWNLOADER_MIDDLEWARES = {
+#    'locations.middlewares.MyCustomDownloaderMiddleware': 543,
+#}
 
 # Enable or disable extensions
 # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
diff --git a/locations/spiders/ace_hardware.py b/locations/spiders/ace_hardware.py
--- a/locations/spiders/ace_hardware.py
+++ b/locations/spiders/ace_hardware.py
@@ -9,12 +9,13 @@
 
 class AceHardwareSpider(scrapy.Spider):
     name = "ace_hardware"
-    item_attributes = { 'brand': "Ace Hardware" }
+    item_attributes = {'brand': "Ace Hardware", 'brand_wikidata': 'Q4672981'}
     allowed_domains = ["www.acehardware.com"]
-    download_delay = 0.1
+    download_delay = 0.7
     start_urls = (
         'https://www.acehardware.com/store-directory',
     )
+    user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
 
     def parse_hours(self, lis):
         o = OpeningHours()
@@ -22,15 +23,10 @@ def parse_hours(self, lis):
         for day in DAY_MAPPING:
             d = day.title()[:2]
 
-            label = lis[day]['label']
-            if label == '0000 - 0000':
+            if lis[day]['label'] == '0000 - 0000':
                 continue
 
-            start, end = label.split(' - ')
-            start = '%s:%s' % (start[:2], start[2:])
-            end = '%s:%s' % (end[:2], end[2:])
-
-            o.add_range(d, start, end)
+            o.add_range(d, lis[day]['openTime'], lis[day]['closeTime'])
         return o.as_opening_hours()
 
     def parse_store(self, response):
@@ -50,8 +46,8 @@ def parse_store(self, response):
             'postcode': store_data['StoreZipCd'],
             'ref': store_data['StoreNumber'],
             'website': response.url,
-            'lat': float(store_data['Latitude']),
-            'lon': float(store_data['Longitude']),
+            'lat': store_data['Latitude'],
+            'lon': store_data['Longitude'],
         }
 
         hours = self.parse_hours(store_data['RegularHours'])


Test patch: