Repository: alltheplaces/alltheplaces

Issue:
Food Warehouse spider missing a handful of stores
The spider for "The Food Warehouse" at https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/the_food_warehouse_gb.py appears to be missing a small number (probably around 8) stores. These include:

* https://www.thefoodwarehouse.com/store-locator/exeter
* https://www.thefoodwarehouse.com/store-locator/falkirk
* https://www.thefoodwarehouse.com/store-locator/Norwich-Longwater
* https://www.thefoodwarehouse.com/store-locator/york-foss-islands

The spider gets the store details from https://www.thefoodwarehouse.com/assets/foodwarehouse/ajax/ . The above stores are all listed there, and I can't see any obvious problems with the data for them, but they're not returned in the dataset at https://alltheplaces-data.openaddresses.io/runs/2024-04-13-13-32-00/output/the_food_warehouse_gb.geojson . The number of stores returned by the spider appears stable, and it looks like the stores above are consistently absent from the returned datasets.

There's nothing in the error log to indicate any problems. I don't know what it signifies, but the stats at https://alltheplaces-data.openaddresses.io/runs/2024-04-13-13-32-00/stats/the_food_warehouse_gb.json say item_dropped_count=8 and 
item_dropped_reasons_count/DropItem=8. I'm guessing that these could be the missing stores, but I don't know why they've been 'dropped'.

There are currently 187 items in the JSON at https://www.thefoodwarehouse.com/assets/foodwarehouse/ajax/ . 5 include "coming soon" in the name, so (correctly) aren't returned by the spider. There's another two that include "now open", so may have been "coming soon" at the last run. That would leave 180 stores to be returned. The number actually returned at the last run was 172, which probably matches the 8 described as being dropped in the stats.

Can anyone see what's going wrong here, and why these stores aren't being returned?


Discussion:
They are being dropped because they are duplicates (`ref` is a duplicate).

Source issue is easily visable with:
```bash
curl https://www.thefoodwarehouse.com/assets/foodwarehouse/ajax/ | jq '.[].storeNo' -r | sort | uniq -c | sort -n
```
```
      4 1234
      2 1765
      2 1744
      2 1738
      2 1573
      2 1531
      1 null
      ....
```

This is doing some good, eg:

(1531)
https://www.thefoodwarehouse.com/store-locator/lincoln
https://www.thefoodwarehouse.com/store-locator/north-hykeham-lincoln

(1573)
https://www.thefoodwarehouse.com/store-locator/north-liverpool
https://www.thefoodwarehouse.com/store-locator/aintree-liverpool

However, as you noted, the existing system doesn't work perfectly, as far as I can tell 2 legitimate stores have 1765
(1765)
https://www.thefoodwarehouse.com/store-locator/store-locator/Norwich-Longwater
https://www.thefoodwarehouse.com/store-locator/store-locator/Bishop-Auckland

And several with 1234.

We can switch it using the url/slug as the unique id, which will allow your examples through. However it would introduce the 2 duplicates.
... There is also an issue with several of them having `https://www.thefoodwarehouse.com/store-locator/default-store`

```bash
curl 'https://www.thefoodwarehouse.com/assets/foodwarehouse/ajax/' | jq '.[] | select(.url=="/store-locator/default-store")'
```

```json
{
  "validFrom": "2019-01-01T12:00:00.000Z",
  "validTo": "2100-12-12T12:00:00.000Z",
  "storeNo": "1737",
  "lat": "53.259390",
  "store-image": "https://assets.iceland.co.uk/i/iceland/store",
  "opening-times": "<p>Mon to Sat: 8am-8pm <br /> Sun: 10am-4pm</p>",
  "lng": "-2.126090",
  "url": "/store-locator/default-store",
  "store-info": "<ul>\n<li>Customer info point</li>\n<li>Wheelchair access</li>\n<li>Wheelchair trolley</li>\n<li>Free customer parking</li>\n</ul>",
  "name": "Macclesfield",
  "store-number": "01625 420972",
  "address": "<p>Lyme Green Retail Park, Macclesfield, Cheshire,<br />\nSK11 OTB</p>"
}
{
  "validFrom": "2021-04-06T08:20:01.390Z",
  "validTo": "2100-12-12T12:00:00.000Z",
  "storeNo": null,
  "lat": "50.615980",
  "lng": "-2.457280",
  "store-image": "https://assets.iceland.co.uk/i/iceland/store",
  "store-info": "<ul><li>Customer info point</li><li>Wheelchair access</li><li>Wheelchair trolley</li><li>Free customer parking</li></ul>",
  "store-number": "01234 567890897",
  "opening-times": "<p>Mon to Sat: 8am-8pm<br />\nSun: 10:00am-4:00pm</p>",
  "name": "Weymouth",
  "url": "/store-locator/default-store",
  "address": "<p>Unit 1 Jubilee Retail Park,<br />\nWeymouth,<br />\nDorset,<br />\nDT4 7BG</p>"
}
{
  "validFrom": "2021-03-08T04:29:00.000Z",
  "validTo": "2100-12-12T12:00:00.000Z",
  "storeNo": "1728",
  "lat": "54.470770",
  "url": "/store-locator/default-store",
  "store-info": "<ul><li>Customer info point</li><li>Wheelchair access</li><li>Wheelchair trolley</li><li>Free customer parking</li></ul>",
  "lng": "-0.596320",
  "name": "Whitby",
  "store-image": "https://assets.iceland.co.uk/i/iceland/store",
  "address": "<p>Unit B,\nWhitby Retail Park,\nWhitby<br />\nYO22 4PU</p>",
  "opening-times": "<p>Mon to Sat: 8am-8pm<br />\nSun: 10am-4pm</p>",
  "store-number": "01947 821607"
}
```

I feel like there is no good option. We could disable de duplication, AFAIK it's only 2 duplicates at present, and they have similar coords and the same postcode so you could pick them out.

Resolving patch:
diff --git a/locations/spiders/the_food_warehouse_gb.py b/locations/spiders/the_food_warehouse_gb.py
--- a/locations/spiders/the_food_warehouse_gb.py
+++ b/locations/spiders/the_food_warehouse_gb.py
@@ -1,8 +1,9 @@
-from scrapy import Spider
+from scrapy import Selector, Spider
 
 from locations.categories import Categories
 from locations.dict_parser import DictParser
 from locations.hours import OpeningHours
+from locations.pipelines.address_clean_up import merge_address_lines
 
 
 class TheFoodWarehouseGBSpider(Spider):
@@ -14,18 +15,18 @@ class TheFoodWarehouseGBSpider(Spider):
     }
     allowed_domains = ["www.thefoodwarehouse.com"]
     start_urls = ["https://www.thefoodwarehouse.com/assets/foodwarehouse/ajax/"]
+    no_refs = True  # https://github.com/alltheplaces/alltheplaces/issues/8237
 
     def parse(self, response):
         for store in response.json():
             item = DictParser.parse(store)
             if "CLOSED" in item["name"].upper() or "COMING SOON" in item["name"].upper():
                 continue
-            item["ref"] = store["storeNo"]
-            item["website"] = "https://www.thefoodwarehouse.com" + store["url"]
+            if store["url"] != "/store-locator/default-store":
+                item["website"] = "https://www.thefoodwarehouse.com" + store["url"]
+            item["branch"] = item.pop("name").removesuffix(" - Now Open")
             item["phone"] = store.get("store-number")
-            item["addr_full"] = (
-                item["addr_full"].replace("<br>", "").replace("<br />", "").replace("<p>", "").replace("</p>", "")
-            )
+            item["addr_full"] = merge_address_lines(Selector(text=item["addr_full"]).xpath("//text()").getall())
             item["opening_hours"] = OpeningHours()
             item["opening_hours"].add_ranges_from_string(store.get("opening-times", ""))
             yield item


Test patch: