Repository: alltheplaces/alltheplaces

Issue:
Spider thales_fr is broken
During the global build at 2021-05-26-14-42-23, spider **thales_fr** failed with **0 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/logs/thales_fr.log) and [the output](https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/thales_fr.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-05-26-14-42-23/output/thales_fr.geojson))


Discussion:
During the global build at 2021-06-30-14-42-26, spider **thales_fr** failed with **0 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/logs/thales_fr.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/thales_fr.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-06-30-14-42-26/output/thales_fr.geojson))
During the global build at 2021-07-14-14-42-22, spider **thales_fr** failed with **0 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/logs/thales_fr.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/thales_fr.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-14-14-42-22/output/thales_fr.geojson))
During the global build at 2021-07-21-14-42-39, spider **thales_fr** failed with **0 features** and **1 errors**.

Here's [the log](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/logs/thales_fr.txt) and [the output](https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/thales_fr.geojson) ([on a map](https://data.alltheplaces.xyz/map.html?show=https://data.alltheplaces.xyz/runs/2021-07-21-14-42-39/output/thales_fr.geojson))

Resolving patch:
diff --git a/locations/spiders/thales_fr.py b/locations/spiders/thales_fr.py
--- a/locations/spiders/thales_fr.py
+++ b/locations/spiders/thales_fr.py
@@ -1,6 +1,8 @@
 # -*- coding: utf-8 -*-
+import io
 import json
 import re
+import zipfile
 
 import scrapy
 
@@ -10,41 +12,50 @@
 
 class ThalesFrSpider(scrapy.Spider):
     name = "thales_fr"
-    item_attributes = {'brand': "Thales"}
-    allowed_domains = ['google.com']
-    start_urls = [
-        'https://www.google.com/maps/d/u/0/embed?mid=1yA7rNTU7COi4W6wrLiZ8Ro0Kem8&ll=48.40753354596532%2C-4.58267066250005&z=6',
-    ]
+    item_attributes = {"brand": "Thales"}
+    allowed_domains = ["google.com"]
+    start_urls = ["https://www.google.com/maps/d/kml?mid=zOpSQcuS01RE.kLfnXlw7pPcM"]
 
     def parse(self, response):
-        data = response.xpath(
-            '//script[@type="text/javascript" and contains(text(), "_pageData")]/text()').extract_first()
-
-        places = re.findall(r"Entity(.*?)KO", data)
-
-        for place in places:
-            addr = re.search(r'Address\\",\[\\"(.*?)\\"\]', place).group(1)
-            if "France" not in addr:
-                pass
+        kmz = zipfile.ZipFile(io.BytesIO(response.body))
+        kml = kmz.open(kmz.filelist[0]).read()
+        data = scrapy.Selector(text=kml, type="xml")
+        data.remove_namespaces()
+        for ref, placemark in enumerate(data.xpath("//Placemark")):
+            yield self.parse_location(ref, placemark)
+
+    def parse_location(self, ref, placemark):
+        addr = re.search(
+            "Address: (.*?)<br>",
+            placemark.xpath(".//description/text()").extract_first(),
+        )[1]
+        if "France" not in addr:
+            addr_full = addr
+            street = locality = postal = ""
+        else:
+            addr_split = addr.split(",")
+            if len(addr_split) == 4:
+                street = addr_split[0] + addr_split[1]
+                postal = re.search(r"\d{5}", addr_split[2]).group(0)
+                locality = ""
             else:
-                addr_split = addr.split(",")
-                if len(addr_split) == 4:
-                    street = addr_split[0] + addr_split[1]
-                    postal = re.search(r'\d{5}', addr_split[2]).group(0)
-                else:
-                    street = addr_split[0]
-                    postal = re.search(r'\d{5}', addr_split[1]).group(0)
-                    locality = addr_split[1].replace(postal, "").strip()
-
-            properties = {
-                'ref': re.search(r'null,(\d*?)\]', place).group(1),
-                'name': re.search(r'\[\\"(.*?)\\"\]', place).group(1).replace("\\\\u0026", "&"),
-                'addr_full': street,
-                'city': locality,
-                'postcode': postal,
-                'country': 'FR',
-                'lat': re.search(r'Latitude\\",\[\\"(.*?)\\', place).group(1),
-                'lon': re.search(r'Longitude\\",\[\\"(.*?)\\', place).group(1)
-            }
-
-            yield GeojsonPointItem(**properties)
+                street = addr_split[0]
+                postal = re.search(r"\d{5}", addr_split[1]).group(0)
+                locality = addr_split[1].replace(postal, "").strip()
+
+        x, y, z = (
+            placemark.xpath(".//coordinates/text()").extract_first().strip().split(",")
+        )
+
+        properties = {
+            "ref": ref,
+            "name": placemark.xpath(".//name/text()").extract_first(),
+            "addr_full": street,
+            "city": locality,
+            "postcode": postal,
+            "country": "FR",
+            "lat": y,
+            "lon": x,
+        }
+
+        return GeojsonPointItem(**properties)


Test patch: