Repository: alltheplaces/alltheplaces

Issue:
Shell spider doesn't return any results for Bulgaria
There are around 100 Shell gas stations in Bulgaria but none of them are shown in the latest spider result. They are listed on their Bulgarian website (https://www.shell.bg/motorists/shell-station-locator.html).


Discussion:
The `GeoMeSpider` store finder makes a lot of queries to a common GeoMe API to find locations across the world. This is duplicated for both `shell` and `bp` that are brands using the `GeoMeSpider`.

The `shell` spider is returning a 403 response mid crawl, terminating the crawl before any locations in Europe are found. Reference: https://data.alltheplaces.xyz/runs/2023-09-02-13-32-00/stats/shell.json

The `bp` spider is also returning a 403 response mid crawl. Reference: https://data.alltheplaces.xyz/runs/2023-09-02-13-32-00/stats/bp.json

Perhaps the answer is to try `download_delay=2` or some other number in `storefinders/geo_me.py` in the hope that the 403 response isn't received. And when a 403 response is returned, perhaps it is possible to pause a few minutes (via a handler for the `errback=` parameter) then retry the request.
`AUTOTHROTTLE_ENABLED` should slow down the requests. I don't know if it extends to a few minuets like you question.

https://docs.scrapy.org/en/latest/topics/autothrottle.html#std-setting-AUTOTHROTTLE_ENABLED
I've got a run going (slowly) with a 0.5 autothrottle and will report back results when available (including total crawl time).
Finished in 8945 seconds with no errors, and still the same number of locations returned (should be +10000 locations including Europe but these are missing).

I'm debugging now to see what logic is wrong in the spider to cause so much of the world to be missed.

Resolving patch:
diff --git a/locations/storefinders/geo_me.py b/locations/storefinders/geo_me.py
--- a/locations/storefinders/geo_me.py
+++ b/locations/storefinders/geo_me.py
@@ -1,3 +1,5 @@
+import random
+
 from scrapy import Spider
 from scrapy.http import JsonRequest
 from scrapy.signals import spider_idle
@@ -62,41 +64,42 @@ def parse_bounding_box(self, response):
 
     def start_location_requests(self):
         self.crawler.signals.disconnect(self.start_location_requests, signal=spider_idle)
-        if len(self.locations_found) > 0:
-            first_search_location = self.locations_found.popitem()
-            first_request = JsonRequest(
-                url=self.url_nearest_to_template.format(
-                    self.key, self.api_version, first_search_location[1][0], first_search_location[1][1]
-                ),
-                callback=self.parse_locations,
-            )
-            self.crawler.engine.crawl(first_request)
+        self.crawler.engine.crawl(self.get_next_location())
 
     def parse_locations(self, response):
         for location in response.json()["locations"]:
+            # Remove found location from the list of locations which
+            # are still waiting to be found.
+            if self.locations_found.get(location["id"]):
+                self.locations_found.pop(location["id"])
+
             if location.get("inactive"):
                 continue
+
             location["street_address"] = location.pop("address")
             item = DictParser.parse(location)
             self.extract_hours(item, location)
             yield from self.parse_item(item, location) or []
 
-            # Remove found location from the list of locations which
-            # are still waiting to be found.
-            if self.locations_found.get(location["id"]):
-                self.locations_found.pop(location["id"])
-
         # Get the next location to do a "nearest to" search from.
-        if len(self.locations_found) > 0:
-            next_search_location = self.locations_found.popitem()
-            yield JsonRequest(
-                url=self.url_nearest_to_template.format(
-                    self.key, self.api_version, next_search_location[1][0], next_search_location[1][1]
-                ),
-                callback=self.parse_locations,
-            )
+        yield self.get_next_location()
+
+    def get_next_location(self) -> JsonRequest:
+        if len(self.locations_found) == 0:
+            return
+        next_search_location_id = random.choice(list(self.locations_found))
+        next_search_location_coords = self.locations_found[next_search_location_id]
+        self.locations_found.pop(next_search_location_id)
+        return JsonRequest(
+            url=self.url_nearest_to_template.format(
+                self.key, self.api_version, next_search_location_coords[0], next_search_location_coords[1]
+            ),
+            callback=self.parse_locations,
+            dont_filter=True,
+        )
 
-    def extract_hours(self, item, location):
+    @staticmethod
+    def extract_hours(item: Feature, location: dict):
         item["opening_hours"] = OpeningHours()
         if location.get("open_status") == "twenty_four_hour":
             item["opening_hours"].add_days_range(DAYS, "00:00", "23:59")


Test patch: