Repository: alltheplaces/alltheplaces

Issue:
Spider aldi_ie is broken
During the global build at 2021-05-21-20-39-25, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/logs/aldi_ie.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-21-20-39-25/output/aldi_ie.geojson))


Discussion:
During the global build at 2021-05-24-14-04-37, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/logs/aldi_ie.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-24-14-04-37/output/aldi_ie.geojson))
During the global build at 2021-05-26-14-42-23, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/aldi_ie.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/aldi_ie.geojson))
During the global build at 2021-06-02-14-42-40, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/aldi_ie.geojson))
During the global build at 2021-06-09-14-42-21, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/aldi_ie.geojson))
During the global build at 2021-06-16-14-42-20, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/aldi_ie.geojson))
During the global build at 2021-06-23-14-42-18, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/aldi_ie.geojson))
During the global build at 2021-06-30-14-42-26, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/aldi_ie.geojson))
During the global build at 2021-07-07-14-42-19, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/aldi_ie.geojson))
During the global build at 2021-07-14-14-42-22, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/aldi_ie.geojson))
During the global build at 2021-07-21-14-42-39, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/aldi_ie.geojson))
During the global build at 2021-07-28-14-42-16, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/aldi_ie.geojson))
During the global build at 2021-08-04-14-42-45, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/aldi_ie.geojson))
During the global build at 2021-08-11-14-42-19, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/aldi_ie.geojson))
During the global build at 2021-08-18-14-42-26, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/aldi_ie.geojson))
During the global build at 2021-08-25-14-42-15, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/output/aldi_ie.geojson))
During the global build at 2021-09-01-14-42-16, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/output/aldi_ie.geojson))
During the global build at 2021-09-08-14-42-47, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/output/aldi_ie.geojson))
During the global build at 2021-09-15-14-42-44, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/aldi_ie.geojson))
During the global build at 2021-09-22-14-42-27, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-22-14-42-27/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-22-14-42-27/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-22-14-42-27/output/aldi_ie.geojson))
During the global build at 2021-09-29-14-42-48, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-29-14-42-48/output/aldi_ie.geojson))
During the global build at 2021-10-06-14-42-44, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-06-14-42-44/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-06-14-42-44/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-06-14-42-44/output/aldi_ie.geojson))
During the global build at 2021-10-13-14-42-23, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-13-14-42-23/output/aldi_ie.geojson))
During the global build at 2021-10-20-14-42-48, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-20-14-42-48/output/aldi_ie.geojson))
During the global build at 2021-10-27-14-42-46, spider **aldi_ie** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-10-27-14-42-46/logs/aldi_ie.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-10-27-14-42-46/output/aldi_ie.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-10-27-14-42-46/output/aldi_ie.geojson))

Resolving patch:
diff --git a/locations/spiders/aldi_ie.py b/locations/spiders/aldi_ie.py
deleted file mode 100644
--- a/locations/spiders/aldi_ie.py
+++ /dev/null
@@ -1,57 +0,0 @@
-# -*- coding: utf-8 -*-
-import scrapy
-import re
-import json
-
-from locations.items import GeojsonPointItem
-
-
-class AldiIESpider(scrapy.Spider):
-    name = "aldi_ie"
-    item_attributes = {"brand": "Aldi"}
-    allowed_domains = ["www.aldi.ie"]
-    start_urls = ("https://www.aldi.ie/sitemap/store",)
-
-    def parse(self, response):
-        response.selector.remove_namespaces()
-        city_urls = response.xpath("//url/loc/text()").extract()
-        for path in city_urls:
-            yield scrapy.Request(
-                path.strip(),
-                callback=self.parse_store,
-            )
-        else:
-            pass
-
-    def parse_store(self, response):
-        json_data = (
-            response.xpath('//script[@type="text/javascript"]/text()')
-            .extract_first()
-            .replace("\n", "")
-            .replace("\t", "")
-            .split(".push(")[1]
-            .rstrip(")")
-        )
-        data = json.loads(json_data)
-        geojson_data = response.xpath(
-            '//script[@class="js-store-finder-initial-state"][@type="application/json"]/text()'
-        ).extract_first()
-        geodata = json.loads(geojson_data)
-
-        properties = {
-            "name": data["seoData"]["name"],
-            "ref": data["seoData"]["name"],
-            "addr_full": data["seoData"]["address"]["streetAddress"],
-            "city": data["seoData"]["address"]["addressLocality"],
-            "postcode": data["seoData"]["address"]["postalCode"],
-            "country": data["seoData"]["address"]["addressCountry"],
-            "website": response.request.url,
-            "opening_hours": str(data["seoData"]["openingHours"])
-            .replace("[", "")
-            .replace("]", "")
-            .replace("'", ""),
-            "lat": float(geodata["store"]["latlng"]["lat"]),
-            "lon": float(geodata["store"]["latlng"]["lng"]),
-        }
-
-        yield GeojsonPointItem(**properties)
diff --git a/locations/spiders/aldi_uk.py b/locations/spiders/aldi_uk.py
--- a/locations/spiders/aldi_uk.py
+++ b/locations/spiders/aldi_uk.py
@@ -1,70 +1,25 @@
 # -*- coding: utf-8 -*-
 import json
-import csv
-import scrapy
 import re
 
 from locations.items import GeojsonPointItem
+from scrapy.spiders import SitemapSpider
 
-COOKIES = {
-    "bm_sz": "04B124C1C96D68082A9F61BAAAF0B6D5~YAAQdjsvF22E8Xl6AQAACr1VfAxPEt+enarZyrOZrBaNvyuX71lK5QPuDR/FgDEWBZVMRhjiIf000W7Z1PiAjxobrz2Y5LcYMH3CvUNvpdS3MjVLUMGwMEBCf9L5nD5Gs9ho2YL8T7Tz7lYvpolvaOlJnKrHyhCFxxk/uyBZ2G/0QrGKLwSaCQShDsz7ink=",
-    "_abck": "440E40C406E69413DCCC08ABAA3E9022~-1~YAAQdjsvF26E8Xl6AQAACr1VfAYznoJdJhX7TNIZW1Rfh6qRhzquXg+L1TWoaL7nZUjXlNls2iPIKFQrCdrWqY/CNXW+mHyXibInMflIXJi5VVB/Swq53kABYJDuXYSlCunYvJAzMSr1q12NOYswz134Y8HRNzVWhkb2jMS5whmHxS/v0vniIvS1TQtKjEQlMGzQYmN41CmLX0JobipQhDtUB4VyNwztb2DCAZiqDX8BLwWg7h/DtPd4158qU69hNhayFTgWmD76/MiR8/T536tMmcoRyWLl4fEtP/XUmKOcksuZO7dbfNxXBffTxIXPYwf1eO77LNuZTCQq5kfsGZLJX8ODju2KSjnIF1vdnyHAe98FDIm+hw==~-1~-1~-1",
-}
 
-HEADERS = {
-    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
-    "accept-encoding": "gzip, deflate, br",
-    "cache-control": "max-age=0",
-    "referer": "https://www.aldi.co.uk/store-finder",
-    "sec-ch-ua": '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"',
-    "sec-ch-ua-mobile": "?0",
-    "sec-fetch-dest": "document",
-    "sec-fetch-mode": "navigate",
-    "sec-fetch-site": "same-origin",
-    "sec-fetch-user": "?1",
-    "upgrade-insecure-requests": "1",
-    "user-agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36",
-}
-
-
-class AldiUKSpider(scrapy.Spider):
+class AldiUKSpider(SitemapSpider):
     name = "aldi_uk"
     item_attributes = {"brand": "Aldi", "brand_wikidata": "Q41171672"}
-    allowed_domains = ["aldi.co.uk"]
+    allowed_domains = ["aldi.co.uk", "aldi.ie"]
     download_delay = 1.5
+    sitemap_urls = [
+        "https://www.aldi.co.uk/sitemap/store-en_gb-gbp",
+        "https://www.aldi.ie/sitemap/store-en_ie-eur",
+    ]
     custom_settings = {
-        "USER_AGENT": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
+        "USER_AGENT": "Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0"
     }
 
-    def start_requests(self):
-        url = "https://www.aldi.co.uk/sitemap/store-en_gb-gbp"
-
-        yield scrapy.http.FormRequest(
-            url=url,
-            method="GET",
-            dont_filter=True,
-            cookies=COOKIES,
-            headers=HEADERS,
-            callback=self.parse,
-        )
-
     def parse(self, response):
-        response.selector.remove_namespaces()
-        store_urls = response.xpath("//url/loc/text()").extract()
-
-        for store_url in store_urls:
-
-            yield scrapy.http.FormRequest(
-                url=store_url,
-                method="GET",
-                dont_filter=True,
-                cookies=COOKIES,
-                headers=HEADERS,
-                callback=self.parse_store,
-            )
-
-    def parse_store(self, response):
-
         store_js = response.xpath(
             '//script[@type="text/javascript"]/text()'
         ).extract_first()
@@ -78,7 +33,7 @@ def parse_store(self, response):
 
         properties = {
             "name": data["seoData"]["name"],
-            "ref": response.request.url.replace("https://www.aldi.co.uk/store/", ""),
+            "ref": geodata["store"]["code"],
             "street_address": data["seoData"]["address"]["streetAddress"],
             "city": data["seoData"]["address"]["addressLocality"],
             "postcode": data["seoData"]["address"]["postalCode"],


Test patch: