Repository: alltheplaces/alltheplaces

Issue:
"TotalEnergies.fr" places that don't exist (Chile)
hello, i came across this project while reading [something](https://notes.labordata.bunkum.us/2023/01/09/corp-registry.html) and noticed that there seems to be a massive quantity of "totalenergies.fr" stores all around chile that don't actually exist.

the company totalenergies "is a multi-energy company that produces and markets fuels, natural gas and electricity" ([source](https://totalenergies.com/company/identity)) but most of chile has a monopoly of very few companies that deal with fuel/electricity and i've never heard or seen totalenergies.fr.

while trying to figure out where this data is coming from, i saw the [`total_energies.py` crawler](https://github.com/alltheplaces/alltheplaces/blame/master/locations/spiders/total_energies.py) and ended up going to https://store.totalenergies.fr/ where the main map is shown, only to notice that there aren't any stores in chile (labelled "Chili" because french). i don't know where they're coming from.

while clicking around the map, i can notice that most of the stores have "name" tags that match to small local towns so i can only assume they're selling totalenergy products (like what?) but this is not very convincing. additionally, i've checked many places where i've been and i can confirm that the store "totalenergies.fr" does NOT exist. the only thing they have in common is that they all seem to be brand-less small local stores for auto repair tools or car products. the openstreetmap tag catalogs every single place as fuel, which is incorrect.

random examples:
- [-70.586128,-33.410666](https://www.alltheplaces.xyz/map/#17.15/-33.410666/-70.586128) "Derco S.A. Comercial automotriz": derco is a company that sells new cars (https://www.dercocenter.cl/). it was founded in chile about 70 years ago and its not related to totalenergies.fr.
- [-70.729637,-33.360176](https://www.alltheplaces.xyz/map/#14.76/-33.36018/-70.72964): this is a local mechanic shop. [heres a pic ](https://user-images.githubusercontent.com/38121691/213894986-655fa9f5-ae1c-4f39-9ded-9124747264f6.png) from google streetview.
- [-71.652532,-35.429484](https://www.alltheplaces.xyz/map/#14.25/-35.42948/-71.65253): a small shop that sells car parts. again, [here's a google streetview picture](https://user-images.githubusercontent.com/38121691/213895065-f92619b7-f772-4f07-80ac-30cf878d2e0c.png).

i checked six different cities and i can confirm that there has actually existed a shop at that place at some point but its most certainly not totalenergies.fr and most of them aren't open anymore. all of them are related to mechanics or cars. the name tag in them is mostly correct, but a lot of them have a person's name (owner, maybe?).

i thought i'd raise an issue about this since the map is polluted with places that don't correspond to the brand. i checked the git log to scrape some emails and ask the spiders/total_energies.py contributors directly but that just feels wrong. i don't want to bother people directly.

i don't have a solution, sorry.



Discussion:
by the way, apologies if my writing is weird or wrong. i haven't used english in a while and i'm a bit rusty on it. also i can't read french so i have no idea what https://www.totalenergies.fr/ is about. i don't know if this repo is the right place to raise an issue about this kind of thing, either. please point me to the right place if this is not it. thanks

Looking at https://store.totalenergies.fr/ and glancing over the code in [here](https://github.com/alltheplaces/alltheplaces/blob/master/locations/storefinders/woosmap.py) and [here](https://github.com/alltheplaces/alltheplaces/blob/master/locations/spiders/total_energies.py), it seems the spider could certainly be improved. For example, we probably shouldn’t tag a place as a gas station when Total’s website doesn’t say so. Also, the spider could emit tags for the kind of gas being sold.

Do you want to try fixing it? Instructions for getting started are [here](https://github.com/alltheplaces/alltheplaces#getting-started).
This may be related to the "brand" https://github.com/alltheplaces/alltheplaces/pull/4254#issuecomment-1326449224

Resolving patch:
diff --git a/locations/categories.py b/locations/categories.py
--- a/locations/categories.py
+++ b/locations/categories.py
@@ -177,6 +177,38 @@ class Fuel(Enum):
     LH2 = "fuel:LH2"
     # Additives
     ADBLUE = "fuel:adblue"
+    ENGINE_OIL = "fuel:engineoil"
+    # Planes
+    AV91UL = "fuel:91UL"
+    AV100LL = "fuel:100LL"
+    AVAUTO_GAS = "fuel:autogas"
+    AVJetA1 = "fuel:JetA1"
+
+    KEROSENE = "fuel:kerosene"
+
+
+class Extras(Enum):
+    ATM = "atm"
+    CAR_WASH = "car_wash"
+    OIL_CHANGE = "service:vehicle:oil_change"
+    TOILETS = "toilets"
+    WHEELCHAIR = "wheelchair"
+    WIFI = "internet_access=wlan"
+
+
+class PaymentMethods(Enum):
+    AMERICAN_EXPRESS = "payment:american_express"
+    APPLE_PAY = "payment:apple_pay"
+    CASH = "payment:cash"
+    CONTACTLESS = "payment:contactless"
+    CREDIT_CARDS = "payment:credit_cards"
+    DEBIT_CARDS = "payment:debit_cards"
+    GOOGLE_PAY = "payment:google_pay"
+    GIROCARD = "payment:girocard"
+    MAESTRO = "payment:maestro"
+    MASTER_CARD = "payment:mastercard"
+    VISA = "payment:visa"
+    V_PAY = "payment:v_pay"
 
 
 def apply_yes_no(attribute, item: Feature, state: bool, apply_positive_only: bool = True):
@@ -195,5 +227,8 @@ def apply_yes_no(attribute, item: Feature, state: bool, apply_positive_only: boo
         tag_key = attribute.value
     else:
         raise AttributeError("string or Enum required")
-    tag_value = "yes" if state else "no"
+    if "=" in tag_key:
+        tag_key, tag_value = tag_key.split("=")
+    else:
+        tag_value = "yes" if state else "no"
     apply_category({tag_key: tag_value}, item)
diff --git a/locations/pipelines.py b/locations/pipelines.py
--- a/locations/pipelines.py
+++ b/locations/pipelines.py
@@ -215,32 +215,33 @@ def process_item(self, item, spider):  # noqa: C901
         check_field(item, spider, "country", (str,))
         check_field(item, spider, "brand", (str,))
 
-        if lat := item.get("lat"):
-            try:
-                lat = float(lat)
-                if not (self.min_lat < lat < self.max_lat):
+        if not item.get("geometry"):
+            if lat := item.get("lat"):
+                try:
+                    lat = float(lat)
+                    if not (self.min_lat < lat < self.max_lat):
+                        spider.crawler.stats.inc_value("atp/field/lat/invalid")
+                    if math.fabs(lat) < 0.01:
+                        spider.crawler.stats.inc_value("atp/field/lat/invalid")
+                except:
+                    lat = None
                     spider.crawler.stats.inc_value("atp/field/lat/invalid")
-                if math.fabs(lat) < 0.01:
-                    spider.crawler.stats.inc_value("atp/field/lat/invalid")
-            except:
-                lat = None
-                spider.crawler.stats.inc_value("atp/field/lat/invalid")
-            item["lat"] = lat
-        else:
-            spider.crawler.stats.inc_value("atp/field/lat/missing")
-        if lon := item.get("lon"):
-            try:
-                lon = float(lon)
-                if not (self.min_lon < lon < self.max_lon):
-                    spider.crawler.stats.inc_value("atp/field/lon/invalid")
-                if math.fabs(lon) < 0.01:
+                item["lat"] = lat
+            else:
+                spider.crawler.stats.inc_value("atp/field/lat/missing")
+            if lon := item.get("lon"):
+                try:
+                    lon = float(lon)
+                    if not (self.min_lon < lon < self.max_lon):
+                        spider.crawler.stats.inc_value("atp/field/lon/invalid")
+                    if math.fabs(lon) < 0.01:
+                        spider.crawler.stats.inc_value("atp/field/lon/invalid")
+                except:
+                    lon = None
                     spider.crawler.stats.inc_value("atp/field/lon/invalid")
-            except:
-                lon = None
-                spider.crawler.stats.inc_value("atp/field/lon/invalid")
-            item["lon"] = lon
-        else:
-            spider.crawler.stats.inc_value("atp/field/lon/missing")
+                item["lon"] = lon
+            else:
+                spider.crawler.stats.inc_value("atp/field/lon/missing")
 
         if twitter := item.get("twitter"):
             if not isinstance(twitter, str):
diff --git a/locations/spiders/total_energies.py b/locations/spiders/total_energies.py
--- a/locations/spiders/total_energies.py
+++ b/locations/spiders/total_energies.py
@@ -1,20 +1,111 @@
-from locations.categories import Categories
+from locations.categories import Categories, Extras, Fuel, PaymentMethods, apply_category, apply_yes_no
 from locations.storefinders.woosmap import WoosmapSpider
 
 
 class TotalEnergiesSpider(WoosmapSpider):
     name = "totalenergies"
-    item_attributes = {
-        "brand": "TotalEnergies",
-        "brand_wikidata": "Q154037",
-        "extras": Categories.FUEL_STATION.value,
-    }
     key = "mapstore-prod-woos"
-    custom_settings = {
-        "DEFAULT_REQUEST_HEADERS": {"Origin": "https://totalenergies.com"},
+    custom_settings = {"DEFAULT_REQUEST_HEADERS": {"Origin": "https://totalenergies.com"}}
+
+    BRANDS = {
+        "tot": {"brand": "Total", "brand_wikidata": "Q154037"},
+        "cepsa": {"brand": "Cepsa", "brand_wikidata": "Q608819"},
+        "aral": {"brand": "Aral", "brand_wikidata": "Q565734"},
+        "bp": {"brand": "BP", "brand_wikidata": "Q152057"},
+        #   1454 "unb"
+        "totalerg": {"brand": "TotalEnergies", "brand_wikidata": "Q154037"},
+        "ela": {"brand": "Elan", "brand_wikidata": "Q57980752"},
+        "mol": {"brand": "MOL", "brand_wikidata": "Q549181"},
+        "tac": {"brand": "Total Access", "brand_wikidata": "Q154037"},
+        "avi": {"brand": "Avia", "brand_wikidata": "Q300147"},
+        "as24": {"brand": "AS 24", "brand_wikidata": "Q2819394"},
+        "ess": {"brand": "Esso", "brand_wikidata": "Q867662"},
+        "agi": {"brand": "Agip", "brand_wikidata": "Q377915"},
+        "westfalen": {"brand": "Westfalen", "brand_wikidata": "Q1411209"},
+        "slovnaft": {"brand": "Slovnaft", "brand_wikidata": "Q1587563"},
+        "she": {"brand": "Shell", "brand_wikidata": "Q110716465"},
+        "totex": {"brand": "Total Express"},
+        #    159 "eos"
+        #    151 "freie"
+        #    112 "dyn"
+        #    106 "gul"
+        #    103 "filt"
+        #     32 "totfa"
+        #     25 "sup"
+        #     20 "wei"
+        #     20 "haa"
+        #     18 "elf"
+        #     15 "omv"
+        #      4 "deal"
+        #      3 "diver"
+        #      1 "g&v"
+        #      1 "fin"
+        #      1 "em"
     }
 
     def parse_item(self, item, feature, **kwargs):
+        if feature["properties"]["user_properties"]["status"] != "2":
+            return
+        if feature["properties"]["user_properties"]["brand"] == "partners":
+            return
+        if feature["properties"]["user_properties"]["poiType"] == "t002":
+            if "be_borne_electrique" in feature["properties"]["tags"]:
+                apply_category(Categories.CHARGING_STATION, item)
+            else:
+                apply_category(Categories.FUEL_STATION, item)
+        else:
+            # Other types, possibly interesting
+            return
+
         item["website"] = f'https://store.totalenergies.fr/en_EN/{item["ref"]}'
 
+        apply_yes_no(Extras.ATM, item, "atm" in feature["properties"]["tags"])
+        apply_yes_no(Extras.CAR_WASH, item, "carwash" in feature["properties"]["tags"])
+        apply_yes_no(Extras.TOILETS, item, "restroom" in feature["properties"]["tags"])
+        apply_yes_no(Extras.WIFI, item, "freewifi" in feature["properties"]["tags"])
+        apply_yes_no(Extras.WHEELCHAIR, item, "accessibility" in feature["properties"]["tags"])
+        apply_yes_no("hgv", item, "truckfriendly" in feature["properties"]["tags"])
+        apply_yes_no(Extras.OIL_CHANGE, item, "oilchange" in feature["properties"]["tags"])
+        apply_yes_no("service:vehicle:glass", item, "carglass" in feature["properties"]["tags"])
+
+        apply_yes_no(Fuel.ENGINE_OIL, item, "engineoil" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.AV100LL, item, "avgas100llautomate" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.AV100LL, item, "avgas100ll" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.AVJetA1, item, "jeta1" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.OCTANE_95, item, "excellium95" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.OCTANE_98, item, "excellium98" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.DIESEL, item, "excelliumdiesel" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.E10, item, "sp95e10" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.ADBLUE, item, "adblue" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.LPG, item, "lpg" in feature["properties"]["tags"])
+        apply_yes_no(Fuel.KEROSENE, item, "kerosene" in feature["properties"]["tags"])
+        # Other fuels?:
+        # gasoil sp95 sp98 super superethanole85 excellium95e10 gasoline92_eg regulargasoline superethanole85
+        # sp95performance happyfuel sp95_lb supereffimax dieseleffimax dieselb10 adbluecar excellium_93
+
+        apply_yes_no(PaymentMethods.AMERICAN_EXPRESS, item, "amex" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.MASTER_CARD, item, "mastercard" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.VISA, item, "visa" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.MAESTRO, item, "maestro" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.V_PAY, item, "vpay" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.GIROCARD, item, "girocard" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.APPLE_PAY, item, "appay" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.GOOGLE_PAY, item, "googlepay" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.CASH, item, "cash" in feature["properties"]["tags"])
+        apply_yes_no(PaymentMethods.CREDIT_CARDS, item, "credit_card" in feature["properties"]["tags"])
+        apply_yes_no("payment:gr_total_card", item, "grcartefr" in feature["properties"]["tags"])
+        apply_yes_no("payment:total_card", item, "totalcard" in feature["properties"]["tags"])
+        apply_yes_no("payment:fleet_fuel_card", item, "ffc" in feature["properties"]["tags"])
+        apply_yes_no("payment:eurotrafic", item, "eurotrafic" in feature["properties"]["tags"])
+        apply_yes_no("payment:club_total_card", item, "carteclubtotal" in feature["properties"]["tags"])
+        # proficard bonjour mpayment westfalencard prepaidcard travelcard ecocash nimbacard ecash sonayacard
+
+        if "shop" in feature["properties"]["tags"]:
+            apply_category(Categories.SHOP_CONVENIENCE, item)
+
+        if brand := self.BRANDS.get(feature["properties"]["user_properties"]["brand"]):
+            item.update(brand)
+        else:
+            item["brand"] = feature["properties"]["user_properties"]["brand"]
+
         yield item


Test patch: