Repository: alltheplaces/alltheplaces

Issue:
Fix Dickey's Barbecue Pit which has now switched to GraphQL
Dickey's Barbecue Pit (dickeys_barbecue_pit) just broke in the latest weekly run. It appears that pages on the website no longer return structured data and instead, the brand has switched over to using GraphQL.

Below is a GraphQL query that returns all store details (first=2000 variable controls how many are returned). The query needs a large clean-up to only select fields necessary, and to provide alternative field names where necessary.

curl 'https://orders-api.dickeys.com/graphql' -X POST -H 'content-type: application/json' --data-raw '{"query":"\n  query Stores(\n    $first: Int\n    $personId: Int\n    $firstAll: Int\n    $sort: [LocationSort]\n    $firstSubcategorySort: [SubcategorySort]\n    $firstSubcategoryFilter: [SubcategoryFilter]\n    $filter: [LocationFilter]\n    $locationHandoffCategoryFilter: [LocationHandoffCategoryFilter]\n    $locationVendorFilter: [LocationVendorFilter]\n  ) {\n    viewer {\n      id\n      personId\n      firstName\n      email\n      locationConnection(first: $first, sort: $sort, filter: $filter) {\n        edges {\n          node {\n            id\n            label\n            slug\n            storeNumber\n            distance\n            active\n            opened\n            isOpened\n            deliveryRadius\n            timezone {\n              id\n              label\n            }\n            locationVendorConnection (filter: $locationVendorFilter) {\n              edges {\n                node {\n                  vendor {\n                    id\n                    label\n                    vendorimage {\n                      url\n                    }\n                  }\n                  deliveryUrl\n                }\n              }\n            }\n            locationExclusiveDeliveryAddressConnection {\n              edges {\n                node {\n                  id\n                  address {\n                    id\n                    address\n                    zip\n                    city\n                  }\n                }\n              }\n            }\n            locationWeekdayConnection(filter: { showHoliday: { eq: true } }) {\n              edges {\n                node {\n                  id\n                  opened\n                  active\n                  closed\n                  locationHours {\n                    id\n                    opened\n                    active\n                    closed\n                  }\n                  weekday {\n                    weekdayId\n                    id\n                    type\n                    label\n                  }\n                }\n              }\n            }\n            locationHandoffCategoryConnection(first: 1000, filter: $locationHandoffCategoryFilter) {\n              edges {\n                node {\n                  id\n                  handoff {\n                    id\n                    label\n                  }\n                  category {\n                    id\n                  }\n                  radius\n                }\n              }\n            }\n            address {\n              id\n              address\n              city\n              state {\n                id\n                abbreviation\n              }\n              zip\n              latitude\n              longitude\n            }\n            phone {\n              id\n              phone\n            }\n            menu {\n              id\n              subcategoryConnection(\n                first: 1\n                sort: $firstSubcategorySort\n                filter: $firstSubcategoryFilter\n              ) {\n                edges {\n                  node {\n                    id\n                    subcategoryId\n                    slug\n                    orderKey\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n      stateConnection(first: $firstAll) {\n        edges {\n          node {\n            id\n            label\n          }\n        }\n      }\n      personConnection(filter: { personId: { eq: $personId } }) {\n        edges {\n          node {\n            id\n            personLocationConnection {\n              edges {\n                node {\n                  id\n                  isFavorite\n                  location {\n                    id\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n","variables":{"first":2000,"firstAll":2147483647,"sort":{"distance":{"latitude":34.1030032,"longitude":-118.4104684,"order":"ASC"}},"firstSubcategorySort":{"orderKey":"ASC"},"firstSubcategoryFilter":{"category":{"categorytype":{"slug":{"eq":"everyday"}}},"dontShowEmpty":{"eq":true}},"filter":{"isOpened":{"eq":true}},"locationHandoffCategoryFilter":[{"active":{"eq":true},"category":{"id":{"eq":"Q2F0ZWdvcnk6MTo5OTk5LTEyLTMxIDIzOjU5OjU5LjAwMDAwMA=="}},"handoff":{"id":{"in":["SGFuZG9mZjoyOjk5OTktMTItMzEgMjM6NTk6NTkuMDAwMDAw","SGFuZG9mZjo0Ojk5OTktMTItMzEgMjM6NTk6NTkuMDAwMDAw"]}}}]}}'

When fixing this spider, it should be renamed to dickeys_barbecue_pit_us as this brand is only present in the USA. There is a single store in Canada now opened, but it appears the intent from the brand is that the Canadian chain would have a different website and different store locator.


Discussion:
Quick go at trimming the query down, `storeTypeId` and  `active` may be interesting.

```graphql
query {
  viewer {
    locationConnection(first: 2000, filter: { isOpened: { eq: true } }) {
      edges {
        node {
          name: label
          slug
          ref: storeNumber
          storeTypeId
          active
          isOpened
          start_date: opened
          end_date: closed
          address {
            street_address: address
            city
            state {
              abbreviation
            }
            zip
            latitude
            longitude
          }
          phone {
            phone
          }
          email
        }
      }
    }
  }
}
```

Resolving patch:
diff --git a/locations/spiders/dickeys_barbecue_pit.py b/locations/spiders/dickeys_barbecue_pit.py
deleted file mode 100644
--- a/locations/spiders/dickeys_barbecue_pit.py
+++ /dev/null
@@ -1,16 +0,0 @@
-from scrapy.spiders import SitemapSpider
-
-from locations.structured_data_spider import StructuredDataSpider
-
-
-class DickeysBarbecuePitSpider(SitemapSpider, StructuredDataSpider):
-    name = "dickeys_barbecue_pit"
-    item_attributes = {"brand": "Dickey's Barbecue Pit", "brand_wikidata": "Q19880747"}
-    allowed_domains = ["dickeys.com"]
-    sitemap_urls = ["https://www.dickeys.com/sitemap.xml"]
-    sitemap_rules = [(r"/locations/[-\w]+/[-\w]+", "parse_sd")]
-    wanted_types = ["Restaurant"]
-
-    def post_process_item(self, item, response, ld_data):
-        item["website"] = response.urljoin(item["website"])
-        yield item
diff --git a/locations/spiders/dickeys_barbecue_pit_us.py b/locations/spiders/dickeys_barbecue_pit_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/dickeys_barbecue_pit_us.py
@@ -0,0 +1,142 @@
+import urllib.parse
+
+from scrapy import Spider
+from scrapy.http import JsonRequest
+
+from locations.dict_parser import DictParser
+from locations.hours import DAYS_FULL, OpeningHours
+
+USA_STATES = {
+    "AK": "Alaska",
+    "AL": "Alabama",
+    "AR": "Arkansas",
+    "AS": "American Samoa",
+    "AZ": "Arizona",
+    "CA": "California",
+    "CO": "Colorado",
+    "CT": "Connecticut",
+    "DC": "District of Columbia",
+    "DE": "Delaware",
+    "FL": "Florida",
+    "GA": "Georgia",
+    "GU": "Guam",
+    "HI": "Hawaii",
+    "IA": "Iowa",
+    "ID": "Idaho",
+    "IL": "Illinois",
+    "IN": "Indiana",
+    "KS": "Kansas",
+    "KY": "Kentucky",
+    "LA": "Louisiana",
+    "MA": "Massachusetts",
+    "MD": "Maryland",
+    "ME": "Maine",
+    "MI": "Michigan",
+    "MN": "Minnesota",
+    "MO": "Missouri",
+    "MS": "Mississippi",
+    "MT": "Montana",
+    "NC": "North Carolina",
+    "ND": "North Dakota",
+    "NE": "Nebraska",
+    "NH": "New Hampshire",
+    "NJ": "New Jersey",
+    "NM": "New Mexico",
+    "NV": "Nevada",
+    "NY": "New York",
+    "OH": "Ohio",
+    "OK": "Oklahoma",
+    "OR": "Oregon",
+    "PA": "Pennsylvania",
+    "PR": "Puerto Rico",
+    "RI": "Rhode Island",
+    "SC": "South Carolina",
+    "SD": "South Dakota",
+    "TN": "Tennessee",
+    "TX": "Texas",
+    "UT": "Utah",
+    "VA": "Virginia",
+    "VI": "Virgin Islands",
+    "VT": "Vermont",
+    "WA": "Washington",
+    "WI": "Wisconsin",
+    "WV": "West Virginia",
+    "WY": "Wyoming",
+}
+
+
+class DickeysBarbecuePitUSSpider(Spider):
+    name = "dickeys_barbecue_pit_us"
+    item_attributes = {"brand": "Dickey's Barbecue Pit", "brand_wikidata": "Q19880747"}
+    allowed_domains = ["orders-api.dickeys.com"]
+    start_urls = ["https://orders-api.dickeys.com/graphql"]
+
+    def start_requests(self):
+        graphql_query = """query {
+    viewer {
+        locationConnection(first: 2000, filter: { isOpened: { eq: true } active: { eq: true } }) {
+            edges {
+                node {
+                    name: label
+                    slug
+                    ref: storeNumber
+                    locationWeekdayConnection(filter: { showHoliday: { eq: true } }) {
+                        edges {
+                            node {
+                                open_time: opened
+                                close_time: closed
+                                weekday {
+                                    day_name: label
+                                }
+                            }
+                        }
+                    }
+                    address {
+                        street_address: address
+                        city
+                        state {
+                            abbreviation
+                        }
+                        zip
+                        latitude
+                        longitude
+                    }
+                    phone {
+                        phone
+                    }
+                    email
+                }
+            }
+        }
+    }
+}"""
+        data = {"query": graphql_query}
+        for url in self.start_urls:
+            yield JsonRequest(url=url, data=data, method="POST")
+
+    def parse(self, response):
+        for location in response.json()["data"]["viewer"]["locationConnection"]["edges"]:
+            location["node"]["address"]["state"] = location["node"]["address"]["state"]["abbreviation"]
+            location["node"]["phone"] = location["node"]["phone"]["phone"]
+            item = DictParser.parse(location["node"])
+            item["lat"] = location["node"]["address"]["latitude"]
+            item["lon"] = location["node"]["address"]["longitude"]
+            item["website"] = (
+                "https://www.dickeys.com/locations/"
+                + USA_STATES[item["state"]].lower()
+                + "/"
+                + urllib.parse.quote(item["city"].lower())
+                + "/"
+                + location["node"]["slug"]
+            )
+            item["opening_hours"] = OpeningHours()
+            for hours_range in location["node"]["locationWeekdayConnection"]["edges"]:
+                if hours_range["node"]["weekday"]["day_name"].title() not in DAYS_FULL:
+                    continue
+                item["opening_hours"].add_range(
+                    hours_range["node"]["weekday"]["day_name"],
+                    hours_range["node"]["open_time"],
+                    hours_range["node"]["close_time"],
+                    "%H:%M:%S",
+                )
+            yield item


Test patch: