Repository: alltheplaces/alltheplaces

Issue:
Spider albert_heijn is broken
During the global build at 2021-06-02-14-42-40, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-02-14-42-40/output/albert_heijn.geojson))


Discussion:
During the global build at 2021-06-09-14-42-21, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-09-14-42-21/output/albert_heijn.geojson))
During the global build at 2021-06-16-14-42-20, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-16-14-42-20/output/albert_heijn.geojson))
During the global build at 2021-06-23-14-42-18, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-23-14-42-18/output/albert_heijn.geojson))
During the global build at 2021-06-30-14-42-26, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/albert_heijn.geojson))
During the global build at 2021-07-07-14-42-19, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-07-14-42-19/output/albert_heijn.geojson))
During the global build at 2021-07-14-14-42-22, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/albert_heijn.geojson))
During the global build at 2021-07-21-14-42-39, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/albert_heijn.geojson))
During the global build at 2021-07-28-14-42-16, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-28-14-42-16/output/albert_heijn.geojson))
During the global build at 2021-08-04-14-42-45, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-04-14-42-45/output/albert_heijn.geojson))
During the global build at 2021-08-11-14-42-19, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-11-14-42-19/output/albert_heijn.geojson))
During the global build at 2021-08-18-14-42-26, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-18-14-42-26/output/albert_heijn.geojson))
During the global build at 2021-08-25-14-42-15, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-08-25-14-42-15/output/albert_heijn.geojson))
During the global build at 2021-09-01-14-42-16, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-01-14-42-16/output/albert_heijn.geojson))
During the global build at 2021-09-08-14-42-47, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-08-14-42-47/output/albert_heijn.geojson))
During the global build at 2021-09-15-14-42-44, spider **albert_heijn** failed with **0 features** and **0 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/logs/albert_heijn.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/albert_heijn.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-09-15-14-42-44/output/albert_heijn.geojson))

Resolving patch:
diff --git a/locations/spiders/albert_heijn.py b/locations/spiders/albert_heijn.py
--- a/locations/spiders/albert_heijn.py
+++ b/locations/spiders/albert_heijn.py
@@ -1,33 +1,53 @@
 # -*- coding: utf-8 -*-
+import json
+import re
+
 import scrapy
+
+from locations.hours import OpeningHours
 from locations.items import GeojsonPointItem
-import json
 
-class AlbertHeijnSpider(scrapy.Spider):
-    name = 'albert_heijn'
-    item_attributes = {'brand': "Albert Heijn"}
-    allowed_domains = ['www.ah.nl']
 
-    def start_requests(self):
-        url = 'https://www.ah.nl/data/winkelinformatie/winkels/json'
-        yield scrapy.Request(url, callback=self.parse)
+class AlbertHeijnSpider(scrapy.Spider):
+    name = "albert_heijn"
+    item_attributes = {"brand": "Albert Heijn", "brand_wikidata": "Q1653985"}
+    allowed_domains = ["www.ah.nl", "www.ah.be"]
+    start_urls = (
+        "https://www.ah.nl/sitemaps/entities/stores/stores.xml",
+        "https://www.ah.be/sitemaps/entities/stores/stores.xml",
+    )
 
     def parse(self, response):
-        stores = json.loads(response.body_as_unicode())
-        for store in stores['stores']:
-            try:
-                phone_number = store['phoneNumber']
-            except:
-                phone_number = ""
-            yield GeojsonPointItem(
-                lat=store['lat'],
-                lon=store['lng'],
-                addr_full="%s %s" % (store['street'], store["housenr"]),
-                city=store['city'],
-                phone=phone_number,
-                state="",
-                postcode=store['zip'],
-                ref=store['no'],
-                country="Netherlands",
-                website="https://www.ah.nl/winkel/albert-heijn/%s/%s/%s" % (store['city'], store['street'], store['no'])
-            )
+        response.selector.remove_namespaces()
+        for url in response.xpath("//loc/text()").extract():
+            if re.search("/winkel/albert-heijn/", url):
+                yield scrapy.Request(url, callback=self.parse_store)
+
+    def parse_store(self, response):
+        for ldjson in response.xpath(
+            '//script[@type="application/ld+json"]/text()'
+        ).extract():
+            data = json.loads(ldjson)
+            if data["@type"] != "GroceryStore":
+                continue
+
+            opening_hours = OpeningHours()
+            for spec in data["openingHoursSpecification"]:
+                opening_hours.add_range(
+                    spec["dayOfWeek"][:2], spec["opens"], spec["closes"]
+                )
+
+            properties = {
+                "ref": response.url,
+                "website": response.url,
+                "name": data["name"],
+                "phone": data["telephone"],
+                "lat": data["geo"]["latitude"],
+                "lon": data["geo"]["longitude"],
+                "addr_full": data["address"]["streetAddress"],
+                "city": data["address"]["addressLocality"],
+                "postcode": data["address"]["postalCode"],
+                "country": data["address"]["addressCountry"],
+                "opening_hours": opening_hours.as_opening_hours(),
+            }
+            yield GeojsonPointItem(**properties)


Test patch: