Repository: alltheplaces/alltheplaces

Issue:
StructuredDataSpider's wanted_types duplicates items
If a JSON-LD contains e.g. `"@type":["HomeGoodsStore","FurnitureStore"]`, then StructuredDataSpider should only consider the item once when given `wanted_types = ["HomeGoodsStore", "FurnitureStore"]`.

(If there's really some reason to handle the item once for each of the types, LinkedDataParser can be invoked directly; if the site contains two distinct items, e.g. "Store" and "Pharmacy", then those are expressed as separate items, not two types on the same item.)


Discussion:
The idea behind it was to be able to get different objects, so "or", I'm open to changes to make it possible to more specifically get items with multiple types "and".

Maybe the end result will be `wanted_types = [["HomeGoodsStore", "FurnitureStore"], "OtherTypeThatWeAlsoWant"]`. I'll think more about this after work.
I kind of agree with @jleedev , the wanted_types thing is a bit of a throwback to my original way of doings things which time has proved rather awry. It was only intended for my parser not to pick up extraneous ld+json records but involves more effort when writing the spider. Effort is something we have been trying to reduce!

So how about filling wanted_types = [] in locations/structured_data_spider.py to be a list of common matches e.g.

wanted_types = ["Store", "LocalBusiness", "HomeGoodsStore", "FurnitureStore", .......]

This would remove the need for setting wanted_types in the spider itself in most instances. It would be nice to somehow let the caller know the matching types that yielded an item, perhaps adding an extras field? This would help if we were applying categories of some type i.e. knowing that a particular store was both "HomeGoodsStore" and "FurnitureStore" is rather useful.

Anyways whatever you decide it's certainly an area that needs a little clean.

I also want to add more callbacks, at lease one with the ld before it's parsed. Probably also extend the current one with the raw ld. I'll start doing this tomorrow.

Resolving patch:
diff --git a/locations/linked_data_parser.py b/locations/linked_data_parser.py
--- a/locations/linked_data_parser.py
+++ b/locations/linked_data_parser.py
@@ -35,8 +35,22 @@ def find_linked_data(response, wanted_type) -> {}:
             if not isinstance(types, list):
                 types = [types]
 
-            for t in types:
-                if LinkedDataParser.check_type(t, wanted_type, default=False):
+            types = [LinkedDataParser.clean_type(t) for t in types]
+
+            if isinstance(wanted_type, list):
+                wanted_types = wanted_type
+            else:
+                wanted_types = [wanted_type]
+
+            wanted_types = [LinkedDataParser.clean_type(t) for t in wanted_types]
+
+            for wanted_type in wanted_types:
+                valid_type = True
+                for t in types:
+                    if not t in wanted_types:
+                        valid_type = False
+
+                if valid_type:
                     return ld_obj
 
     @staticmethod
@@ -173,10 +187,13 @@ def check_type(type: str, wanted_type: str, default: bool = True) -> bool:
         if default and type is None:
             return True
 
+        return LinkedDataParser.clean_type(type) == wanted_type.lower()
+
+    @staticmethod
+    def clean_type(type: str) -> str:
         return (
             type.lower()
             .replace("http://", "")
             .replace("https://", "")
             .replace("schema.org/", "")
-            == wanted_type.lower()
         )
diff --git a/locations/structured_data_spider.py b/locations/structured_data_spider.py
--- a/locations/structured_data_spider.py
+++ b/locations/structured_data_spider.py
@@ -38,7 +38,19 @@ def extract_image(item, response):
 
 class StructuredDataSpider(Spider):
 
-    wanted_types = []
+    wanted_types = [
+        "LocalBusiness",
+        "Store",
+        "Restaurant",
+        "BankOrCreditUnion",
+        "GroceryStore",
+        "FastFoodRestaurant",
+        "Hotel",
+        "Place",
+        "ClothingStore",
+        "DepartmentStore",
+        "HardwareStore",
+    ]
     search_for_email = True
     search_for_phone = True
     search_for_twitter = True
@@ -47,7 +59,10 @@ class StructuredDataSpider(Spider):
     def parse_sd(self, response):
         MicrodataParser.convert_to_json_ld(response)
         for wanted_type in self.wanted_types:
-            if item := LinkedDataParser.parse(response, wanted_type):
+            if ld_item := LinkedDataParser.find_linked_data(response, wanted_type):
+                self.pre_process_data(ld_item)
+
+                item = LinkedDataParser.parse_ld(ld_item)
 
                 if item["ref"] is None:
                     if hasattr(self, "rules"):
@@ -79,8 +94,16 @@ def parse_sd(self, response):
                 if self.search_for_image and item.get("image") is None:
                     extract_image(item, response)
 
-                yield from self.inspect_item(item, response)
+                yield from self.post_process_item(item, response, ld_item)
+
+    def pre_process_data(self, ld_data, **kwargs):
+        """Override with any pre-processing on the item."""
+        pass
+
+    def post_process_item(self, item, response, ld_data, **kwargs):
+        """Override with any post-processing on the item."""
+        yield from self.inspect_item(item, response)
 
     def inspect_item(self, item, response):
-        """Override with any additional processing on the item."""
+        """Deprecated, please use post_process_item(self, item, response, ld_data):"""
         yield item


Test patch:
diff --git a/tests/test_ld_parser.py b/tests/test_ld_parser.py
--- a/tests/test_ld_parser.py
+++ b/tests/test_ld_parser.py
@@ -1,5 +1,7 @@
 import json
 
+from scrapy.http import HtmlResponse
+
 from locations.linked_data_parser import LinkedDataParser
 
 
@@ -232,3 +234,30 @@ def test_check_type():
         is True
     )
     assert LinkedDataParser.check_type("postalAddress", "GeoCoordinates") is False
+
+
+def test_multiple_types():
+    response = HtmlResponse(
+        url="",
+        encoding="utf-8",
+        body="""<script type="application/ld+json">
+                {
+                    "@context": "https://schema.org",
+                    "@type": ["http://schema.org/Place", "Thing"],
+                    "name": "test 1"
+                }
+            </script>
+            <script type="application/ld+json">
+                {
+                    "@context": "https://schema.org",
+                    "@type": "http://schema.org/Place",
+                    "name": "test 2"
+                }
+            </script>""",
+    )
+
+    assert (
+        LinkedDataParser.find_linked_data(response, ["Place", "Thing"])["name"]
+        == "test 1"
+    )
+    assert LinkedDataParser.find_linked_data(response, "Place")["name"] == "test 2"