Repository: alltheplaces/alltheplaces

Issue:
524 allstate insurance agents
Fixes #524 


Discussion:
I ran the spiders in this pull request and got these results:

|Spider|Results|Log|
|---|---|---|
|[`locations/spiders/allstate_insurance_agents.py`](https://github.com/alltheplaces/alltheplaces/blob/7d30f15c7f1994cc3c3ab107580dbaad1c134786/locations/spiders/allstate_insurance_agents.py)|[85 items](https://s3.amazonaws.com/placescraper-results/results/allstate_insurance_agents/2017-12-19-03-29-19/output.geojson) ([Map](https://s3.amazonaws.com/placescraper-results/map.html?show=https://s3.amazonaws.com/placescraper-results/results/allstate_insurance_agents/2017-12-19-03-29-19/output.geojson))|Resulted in a `timeout` ([Log](https://s3.amazonaws.com/placescraper-results/results/allstate_insurance_agents/2017-12-19-03-29-19/log.txt))|


Resolving patch:
diff --git a/locations/spiders/chevron.py b/locations/spiders/chevron.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/chevron.py
@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+import scrapy
+import json
+import urllib
+
+from locations.items import GeojsonPointItem
+
+HEADERS = {'Content-Type': 'application/json'}
+
+WIKIBRANDS = {
+    'Chevron': 'Q319642',
+    'Texaco': 'Q775060'
+}
+
+
+class ChevronSpider(scrapy.Spider):
+    name = "chevron"
+    item_attributes = {'brand': "Chevron", 'brand_wikidata': "Q319642"}
+    allowed_domains = ["www.chevronwithtechron.com"]
+    download_delay = 0.2
+
+    def start_requests(self):
+        with open('./locations/searchable_points/us_centroids_10mile_radius.csv') as points:
+            next(points)  # skip the header row
+            for point in points:
+                row = point.split(',')
+
+                yield scrapy.FormRequest(
+                    url='https://www.chevronwithtechron.com/webservices/ws_getChevronTexacoNearMe_r2.aspx',
+                    method='GET',
+                    formdata={
+                        'brand': 'chevronTexaco',
+                        'radius': '10',
+                        'lat': row[1].strip(),
+                        'lng': row[2].strip()
+                    }
+                )
+
+    def parse(self, response):
+        result = json.loads(response.body_as_unicode())
+
+        if int(result['count']) == 50:
+            self.logger.warning(
+                "received maximum number of stations, reduce the search radius to make sure we don't miss any"
+            )
+
+        for station in result['stations']:
+            yield GeojsonPointItem(
+                lat=station['lat'],
+                lon=station['lng'],
+                name=station['name'],
+                addr_full=station['address'],
+                city=station['city'],
+                state=station['state'],
+                postcode=station['zip'],
+                country='US',
+                phone=station['phone'],
+                website=f"https://www.chevronwithtechron.com/station/id{station['id']}",
+                opening_hours=station['hours'],
+                ref=station['id'],
+                brand=station['brand'],
+                brand_wikidata=WIKIBRANDS[station['brand']],
+                extras={
+                    'amenity': 'fuel',
+                    'diesel': station['diesel'] == '1',
+                    'truck_stop': station['truckstop'] == '1',
+                    'convenience_store': station['cstore'] == '1',
+                    'car_wash': station['carwash'] == '1',
+                    'restroom': station['restroom'] == '1',
+                }
+            )


Test patch: