Repository: alltheplaces/alltheplaces

Issue:
Jacques’ Wein-Depot (DE) (shop/wine) (100+ locations)
### Brand name

Jacques’ Wein-Depot

German wine store chain

### Wikidata ID

Q1678150
https://www.wikidata.org/wiki/Q1678150
https://www.wikidata.org/wiki/Special:EntityData/Q1678150.json

### Store finder url(s)

https://www.jacques.de/weindepots
https://www.jacques.de/sitemap.xml
https://www.jacques.de/weindepot/D025/Bielefeld-Babenhausen

Official Url(s): https://www.jacques.de/
- [x] I have run the below and if anything in generated, added it.
- [x] I have manually checked for a storefinder
- [ ] This is not a manufacturer listing outlets operating under their own brands / this will not create duplicates
`pipenv run scrapy sf --brand-wikidata=Q1678150 https://www.jacques.de/`
```
```

----



Discussion:
1. Search https://www.jacques.de/weindepots for something similar to `<script src="https://www.jacques.de/main.18490de1fe72ffeb.js" type="module"></script>`
2. Load `main.{build_id_that_can_change_over_time}.js`
3. Extract the `xApiKeyCommerce` API key from within (example: `,xApiKeyCommerce:"3ktDIbM06aHMAbGRA1y3NSBpVSuoCUtq",`)
4. Get an authorization bearer token: `curl -X POST -H "x-api-key: {key_extracted_in_step_3}" https://commerce.api.jacques.de/authentication/v1/anonymous`
5. Request list of locations: `curl 'https://commerce.api.jacques.de/depot/v1/locations' -H 'Accept: application/json' -H 'Accept-Language: de-DE' -H 'x-api-key: {key_extracted_in_step_3}' -H 'Authorization: Bearer {token_returned_in_step_4}'`
6. For each location, request details: `curl 'https://commerce.api.jacques.de/depot/v1/depot/D005' -H 'Accept: application/json' -H 'Accept-Language: de-DE' -H 'x-api-key: {key_extracted_in_step_3}' -H 'Authorization: Bearer {token_returned_in_step_4}'`
7. Use `DictParser.parse` and tidy up results. Opening hours are included.

Resolving patch:
diff --git a/locations/dict_parser.py b/locations/dict_parser.py
--- a/locations/dict_parser.py
+++ b/locations/dict_parser.py
@@ -5,6 +5,7 @@ class DictParser:
     ref_keys = [
         "ref",
         "id",
+        "identifier",
         "store-id",
         "store-number",
         "shop-number",
diff --git a/locations/spiders/jacques_wein_depot_de.py b/locations/spiders/jacques_wein_depot_de.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/jacques_wein_depot_de.py
@@ -0,0 +1,88 @@
+from scrapy import Request, Spider
+from scrapy.http import JsonRequest
+from unidecode import unidecode
+
+from locations.categories import Categories
+from locations.dict_parser import DictParser
+from locations.hours import OpeningHours
+
+
+class JacquesWeinDepotDESpider(Spider):
+    name = "jacques_wein_depot_de"
+    item_attributes = {
+        "brand": "Jacques’ Wein-Depot",
+        "brand_wikidata": "Q1678150",
+        "extras": Categories.SHOP_WINE.value,
+    }
+    allowed_domains = ["www.jacques.de", "commerce.api.jacques.de"]
+    start_urls = ["https://www.jacques.de/weindepots"]
+
+    def parse(self, response):
+        main_js_url = "https://www.jacques.de/" + response.xpath('//script[contains(@src, "main.")]/@src').get()
+        yield Request(url=main_js_url, callback=self.parse_api_key)
+
+    def parse_api_key(self, response):
+        api_key = response.text.split('xApiKeyCommerce:"', 1)[1].split('"', 1)[0]
+        yield JsonRequest(
+            url="https://commerce.api.jacques.de/authentication/v1/anonymous",
+            method="POST",
+            headers={"x-api-key": api_key},
+            callback=self.parse_auth_token,
+        )
+
+    def parse_auth_token(self, response):
+        api_key = response.request.headers.get("x-api-key")
+        auth_token = response.json()["token"]
+        headers = {
+            "Accept-Language": "de-DE",
+            "x-api-key": api_key,
+            "Authorization": f"Bearer {auth_token}",
+        }
+        yield JsonRequest(
+            url="https://commerce.api.jacques.de/depot/v1/locations",
+            headers=headers,
+            callback=self.parse_locations_list,
+        )
+
+    def parse_locations_list(self, response):
+        api_key = response.request.headers.get("x-api-key")
+        auth_header_value = response.request.headers.get("Authorization")
+        headers = {
+            "Accept-Language": "de-DE",
+            "x-api-key": api_key,
+            "Authorization": auth_header_value,
+        }
+        for location in response.json():
+            location_id = location["identifier"]
+            yield JsonRequest(
+                url=f"https://commerce.api.jacques.de/depot/v1/depot/{location_id}",
+                headers=headers,
+                callback=self.parse_location,
+            )
+
+    def parse_location(self, response):
+        location = response.json()
+        item = DictParser.parse(location)
+        item["lat"] = location["location"]["coordinates"][0]
+        item["lon"] = location["location"]["coordinates"][1]
+        item["phone"] = location["address"].get("phone")
+        item["email"] = location["address"].get("email")
+        item["website"] = "https://www.jacques.de/weindepot/{}/{}".format(
+            item["ref"], unidecode(item["name"]).replace(" ", "-")
+        )
+
+        item["opening_hours"] = OpeningHours()
+        for day_name, day_hours in location.get("openingHours").items():
+            if not isinstance(day_hours, dict):
+                # Ignore validFrom and validTo fields that occasionally appear
+                continue
+            if "postMeridiem" in day_hours.keys():
+                item["opening_hours"].add_range(
+                    day_name.title(), day_hours["postMeridiem"]["from"], day_hours["postMeridiem"]["to"], "%H%M"
+                )
+            elif "anteMeridiem" in day_hours.keys():
+                item["opening_hours"].add_range(
+                    day_name.title(), day_hours["anteMeridiem"]["from"], day_hours["anteMeridiem"]["to"], "%H%M"
+                )
+
+        yield item


Test patch: