Repository: alltheplaces/alltheplaces

Issue:
AAA Offices
http://www.aaa.com/office/


Discussion:
AAA has locations by zip and radius.
I have a list of all 40,000 zip codes, is it wise to have over 40,000 start_urls
or do you have any recommendations?

So i figured it also allows search by city name e.g
http://tdr.aaa.com/tdrl/search.jsp?searchtype=O&radius=500&format=json&ident=AAACOM&destination=Richmond,%20VA

http://www.aaa.com/office/assets/places.json also provides the city names, which is 33,145 start_urls we are looking at.
i will go for this. thanks
This issue hasn't been worked on in a while. I'm removing the assignment so others can work on it. If you're still working on it, please reassign yourself and reference a pull request or branch where you're doing your work.

Resolving patch:
diff --git a/locations/spiders/aaa.py b/locations/spiders/aaa.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/aaa.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+import json
+
+import scrapy
+
+from locations.items import GeojsonPointItem
+from urllib.parse import urlencode
+
+URL = 'https://tdr.aaa.com/tdrl/search.jsp?'
+
+
+class AAASpider(scrapy.Spider):
+    name = "aaa"
+    item_attributes = {'brand': 'AAA'}
+    allowed_domains = ['tdr.aaa.com']
+    download_delay = 1.0
+
+    def start_requests(self):
+        point_files = [
+            './locations/searchable_points/us_centroids_50mile_radius.csv',
+            './locations/searchable_points/ca_centroids_50mile_radius.csv'
+        ]
+
+        for point_file in point_files:
+            with open(point_file) as points:
+                next(points)  # Ignore the header
+                for point in points:
+                    _, lat, lon = point.strip().split(',')
+
+                    params = {
+                        'searchtype': 'O',
+                        'radius': '5000',
+                        'format': 'json',
+                        'ident': 'AAACOM',
+                        'destination': ','.join([lat, lon])
+                    }
+
+                    yield scrapy.http.Request(URL + urlencode(params), callback=self.parse)
+
+    def parse(self, response):
+        location_data = json.loads(response.text)
+        locations = location_data["aaa"]["services"].get("travelItems")
+
+        # Handle inconsistent json objects
+        if locations and isinstance(locations["travelItem"], list):
+            for location in locations["travelItem"]:
+                properties = {
+                    'ref': location["id"],
+                    'name': location["itemName"],
+                    'addr_full': location["addresses"]["address"]["addressLine"],
+                    'city': location["addresses"]["address"]["cityName"],
+                    'state': location["addresses"]["address"]["stateProv"]["code"],
+                    'postcode': location["addresses"]["address"]["postalCode"],
+                    'country': location["addresses"]["address"]["countryName"]["code"],
+                    'lat': location["position"]["latitude"],
+                    'lon': location["position"]["longitude"],
+                    'phone': location["phones"].get("phone", {}).get("content"),
+                }
+
+                yield GeojsonPointItem(**properties)
+        elif locations:  # if there is data and the response has one location not structured in a list
+            location = locations["travelItem"]
+            properties = {
+                'ref': location["id"],
+                'name': location["itemName"],
+                'addr_full': location["addresses"]["address"]["addressLine"],
+                'city': location["addresses"]["address"]["cityName"],
+                'state': location["addresses"]["address"]["stateProv"]["code"],
+                'postcode': location["addresses"]["address"]["postalCode"],
+                'country': location["addresses"]["address"]["countryName"]["code"],
+                'lat': location["position"]["latitude"],
+                'lon': location["position"]["longitude"],
+                'phone': location["phones"].get("phone", {}).get("content"),
+            }
+
+            yield GeojsonPointItem(**properties)


Test patch: