Repository: alltheplaces/alltheplaces

Issue:
Bi-Mart (US) (Kibo)
### Brand name

Bi-Mart

American retail store chain

### Wikidata ID

Q4902331
https://www.wikidata.org/wiki/Q4902331
https://www.wikidata.org/wiki/Special:EntityData/Q4902331.json

### Store finder url(s)

Official Url(s): http://www.bimart.com/
- [x] I have run the below and if anything in generated, added it.
- [ ] I have manually checked for a storefinder
- [ ] This is not a manufacturer listing outlets operating under their own brands / this will not create duplicates
`pipenv run scrapy sf --brand-wikidata=Q4902331 http://www.bimart.com/`
```
from locations.storefinders.kibo import KiboSpider


class BiMartUSSpider(KiboSpider):
        name = "bimart_us"
        item_attributes = {
                "brand_wikidata": "Q4902331",
                "brand": "Bi-Mart",
        }
        start_urls = [
                "https://www.bimart.com/api/commerce/storefront/locationUsageTypes/SP/locations",
        ]
```



Discussion:
2024-02-29 11:37:16 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.bimart.com/api/commerce/storefront/locationUsageTypes/SP/locations?pageSize=1000> (referer: None)
2024-02-29 11:37:16 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.bimart.com/api/commerce/storefront/locationUsageTypes/SP/locations?pageSize=1000>: HTTP status code is not handled or not allowed
2024-02-29 11:37:16 [scrapy.core.engine] INFO: Closing spider (finished)
2024-02-29 11:37:16 [locations.pipelines.duplicates] INFO: Dropped 0 duplicate items
2024-02-29 11:37:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'atp/cdn/akamai/response_count': 2,
 'atp/cdn/akamai/response_status_count/403': 2,
 'downloader/request_bytes': 684,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 1106,
 'downloader/response_count': 2,
 'downloader/response_status_count/403': 2,
 'elapsed_time_seconds': 1.467369,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2024, 2, 29, 11, 37, 16, 175012, tzinfo=datetime.timezone.utc),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/403': 1,
 'log_count/DEBUG': 3,
 'log_count/INFO': 10,
 'memusage/max': 150167552,
 'memusage/startup': 150167552,
 'response_received_count': 2,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/403': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2024, 2, 29, 11, 37, 14, 707643, tzinfo=datetime.timezone.utc)}
2024-02-29 11:37:16 [scrapy.core.engine] INFO: Spider closed (finished)
Works for me locally.
```
curl 'https://www.bimart.com/api/commerce/storefront/locationUsageTypes/SP/locations?pageSize=1000' \
  -H 'authority: www.bimart.com' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'accept-language: en-GB,en;q=0.9,en-US;q=0.8,de;q=0.7,es;q=0.6,fr;q=0.5' \
  -H 'cache-control: max-age=0' \
  -H 'cookie: sb-sf-at-prod-s=at=9AQlj%2FOkUGeF9XjLQn9yvbxgbagba5PSwAqLQK%2BiT9E3tyDZ3fI%2B4OfZQtvbqyHgXnG66gVATuKDVSWsvGpbON0P2SNEKs9L3fzjmKjFkadpxNdWY3StOwVJJw7Np0c%2F3wO8GXgB7xc9EW9BnhLbKAIJkU%2BbuZuAebhz8uKTpYYRbMhSABG600x2rOABGb0tdVTLVCl%2B4FRxLKGVmTUfQONSPLfaXtRGfJt6VEzDuctoZmTG1K4%2F9C7D7W5knXvwswPeg%2FhxPPHZ1GpBWnPMv73KEBmcCl8wIhlOYUXox8%2BnjzjNBh4EAUOeT8oHP16RO6f%2FWcA%2FKif5RFcYk59M%2Bg%3D%3D&dt=2024-02-29T11%3A38%3A00.6536843Z; sb-sf-at-prod=at=9AQlj%2FOkUGeF9XjLQn9yvbxgbagba5PSwAqLQK%2BiT9E3tyDZ3fI%2B4OfZQtvbqyHgXnG66gVATuKDVSWsvGpbON0P2SNEKs9L3fzjmKjFkadpxNdWY3StOwVJJw7Np0c%2F3wO8GXgB7xc9EW9BnhLbKAIJkU%2BbuZuAebhz8uKTpYYRbMhSABG600x2rOABGb0tdVTLVCl%2B4FRxLKGVmTUfQONSPLfaXtRGfJt6VEzDuctoZmTG1K4%2F9C7D7W5knXvwswPeg%2FhxPPHZ1GpBWnPMv73KEBmcCl8wIhlOYUXox8%2BnjzjNBh4EAUOeT8oHP16RO6f%2FWcA%2FKif5RFcYk59M%2Bg%3D%3D; _mzvr=oavgSyw0I0O9XxjeIksL0Q; _mzvs=nn; ak_bmsc=96EE22039B1D711287C591EB48FD9625~000000000000000000000000000000~YAAQB6N0aDULju6NAQAA6TSo9BZoIq9Q0qSDpjPP9SdIpFypH8IaL/t877VjR1TtuAn+cdzXpxRE6CY8+GnZUdSEtwWg1AtO+V4DRve6yRh8eE7gD63i3xPGKfPSD93OCbYkRjOjsV1ETRRoqvPEpujUeNl1CpMb0TwaULu0NGex7VhlUGBPaggLimG1BtWGOUNBjw3uz1rYFSrqnecijbEFpy0T4mI0RhUwmEIYRR6N1aWdQtixKt/u9oTtcdE94m2TxYq5nkyWFSkG4clZERSirDlgjPdgrlQAag1OZqUmXoNq2YWmxEcQVpzGeg9K/HeXwwecjGhFe5KOypRJHh/j/VI+hd+sWfqKWeH7i5BjsUysFJHsrLKlHkrv9RzepRlTx+S6rGtZdjc=; _mzvt=Xs34ClVDbU2z0sdfEWcqNQ; _mzPc=eyJjb3JyZWxhdGlvbklkIjoiMmFiYTRlZDk1ZWY1NDI1NTlkZGRiMmVkNjU4YjNmMGQiLCJpcEFkZHJlc3MiOiIxNDkuMTY3LjIzLjE5NiIsImlzRGVidWdNb2RlIjpmYWxzZSwiaXNDcmF3bGVyIjpmYWxzZSwiaXNNb2JpbGUiOmZhbHNlLCJpc1RhYmxldCI6ZmFsc2UsImlzRGVza3RvcCI6dHJ1ZSwidmlzaXQiOnsidmlzaXRJZCI6IlhzMzRDbFZEYlUyejBzZGZFV2NxTlEiLCJ2aXNpdG9ySWQiOiJvYXZnU3l3MEkwTzlYeGplSWtzTDBRIiwiaXNUcmFja2VkIjpmYWxzZSwiaXNVc2VyVHJhY2tlZCI6ZmFsc2V9LCJ1c2VyIjp7ImlzQXV0aGVudGljYXRlZCI6ZmFsc2UsInVzZXJJZCI6ImFkMTY2ZTJiZWI1YjQ3YTg5MjQxMjk1Y2FlYWVhZjNjIiwiZmlyc3ROYW1lIjoiIiwibGFzdE5hbWUiOiIiLCJlbWFpbCI6IiIsImlzQW5vbnltb3VzIjp0cnVlLCJiZWhhdmlvcnMiOlsxMDE0XSwiaXNTYWxlc1JlcCI6ZmFsc2V9LCJ1c2VyUHJvZmlsZSI6eyJ1c2VySWQiOiJhZDE2NmUyYmViNWI0N2E4OTI0MTI5NWNhZWFlYWYzYyIsImZpcnN0TmFtZSI6IiIsImxhc3ROYW1lIjoiIiwiZW1haWxBZGRyZXNzIjoiIiwidXNlck5hbWUiOiIifSwiaXNFZGl0TW9kZSI6ZmFsc2UsImlzQWRtaW5Nb2RlIjpmYWxzZSwibm93IjoiMjAyNC0wMi0yOVQxMTozODowMS40NDU5NjlaIiwiY3Jhd2xlckluZm8iOnsiaXNDcmF3bGVyIjpmYWxzZX0sImN1cnJlbmN5UmF0ZUluZm8iOnt9fQ%3D%3D; bm_sv=BBD4335CCF296F2A771E6DDDE7C8E41B~YAAQB6N0aFsQju6NAQAAm3ao9BaYINIVHQvNAq2e6NKz3SsPVEDB2r78XgjPzNsOjrGCltc5s43J6hmLb4UR1AI0/i1iKmYkSc6f4tWUm9mieWivtOVpiljjCTF+Pg5MG2npYA7J/bOw+gkbz1DMkRQ8xsI9XnTt0Xko94PX2wCenyp4WkVUdxZCOW2kBvWXyYq9AVEKzCdIqSniK8lbx+Eh0/P1IwN9o+eF4uEOE7eXJmE2lqeGievd4bxN3Llv~1' \
  -H 'dnt: 1' \
  -H 'if-none-match: "54d40764"' \
  -H 'referer: https://friendly-enigma-65qqgjq5g3x56w.github.dev/' \
  -H 'sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-gpc: 1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36' \
  --compressed
```

Resolving patch:
diff --git a/locations/spiders/bi_mart_us.py b/locations/spiders/bi_mart_us.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/bi_mart_us.py
@@ -0,0 +1,19 @@
+from locations.categories import Categories
+from locations.storefinders.kibo import KiboSpider
+from locations.user_agents import BROWSER_DEFAULT
+
+
+class BiMartUSSpider(KiboSpider):
+    name = "bi_mart_us"
+    item_attributes = {
+        "brand": "Bi-Mart",
+        "brand_wikidata": "Q4902331",
+        "extras": Categories.SHOP_DEPARTMENT_STORE.value,
+    }
+    start_urls = ["https://www.bimart.com/api/commerce/storefront/locationUsageTypes/SP/locations"]
+    user_agent = BROWSER_DEFAULT
+    api_filter = "tenant~siteId eq 49677 and locationType.Code eq MS"
+
+    def parse_item(self, item, location):
+        item.pop("email", None)
+        yield item
diff --git a/locations/storefinders/kibo.py b/locations/storefinders/kibo.py
--- a/locations/storefinders/kibo.py
+++ b/locations/storefinders/kibo.py
@@ -13,9 +13,13 @@
 
 class KiboSpider(Spider):
     page_size = 1000
+    api_filter = None
 
     def start_requests(self):
-        yield JsonRequest(url=f"{self.start_urls[0]}?pageSize={self.page_size}")
+        if self.api_filter:
+            yield JsonRequest(url=f"{self.start_urls[0]}?pageSize={self.page_size}&filter={self.api_filter}")
+        else:
+            yield JsonRequest(url=f"{self.start_urls[0]}?pageSize={self.page_size}")
 
     def parse(self, response, **kwargs):
         for location in response.json()["items"]:


Test patch: