Repository: alltheplaces/alltheplaces

Issue:
Walgreens
https://www.walgreens.com/storelocator/find.jsp?tab=store+locator&requestType=locator


Discussion:
working on this
No progress for 7 days, re-opening this for others.
Have problems
Walgreen's page sends requests to google to get lat/lon and then second request to walgreen's server to get list of near shops (radius only 50miles)
scrapy don't do all these requests
I can use google API to request city positions and then ask list of near shops

They have a "Browse by state" listing: https://www.walgreens.com/storelistings/storesbystate.jsp?requestType=locator
Of course I did
But at final stage they use algorythm I've described
![image](https://user-images.githubusercontent.com/9920034/34075055-501cba74-e270-11e7-9e00-021df3918ef6.png)
![image](https://user-images.githubusercontent.com/9920034/34075078-b81ed288-e270-11e7-8955-c276a73394b7.png)
![image](https://user-images.githubusercontent.com/9920034/34075092-02c1636e-e271-11e7-9993-44dd396325df.png)

I just ask you to let me using google API in spyder 
Or I can prepare table of cities and lan/lon pair to integrate it into spider
Thank you
@vlad555 
please look at this url.
https://customersearch.walgreens.com/storelocator/v1/stores/search
you may can put some static data with big radius and limit.
{"r":"50","lat":"38.5830","lng":"-77.3158","requestType":"locator","s":"15","p":"1"}
here lat and lng will be static.
you can test this on Postman.
@SelfnessAid 
If you checked my 3-rd screenshot I already got this this link and parameters
But Ian said searching by lat/lng with small radius (50 miles in this case) isn't good solution.
In one project I had 9 circles for 800-mile radius.
With 50-mile circles we will have them much more and much more overlapped zones.
As you can see I've  asked about integrating  lat/lon points into spider.
But Ian still not answered
Yep, please hold off on writing these spiders until we come up with a way to efficiently crawl sites like these without tens of thousands of requests.

Resolving patch:
diff --git a/locations/spiders/walgreens.py b/locations/spiders/walgreens.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/walgreens.py
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+import json
+import re
+import scrapy
+from scrapy.selector import Selector
+
+from locations.items import GeojsonPointItem
+from locations.hours import OpeningHours
+
+
+class WalgreensSpider(scrapy.Spider):
+
+    name = "walgreens"
+    allowed_domains = ["www.walgreens.com"]
+    download_delay = 0.1
+    start_urls = (
+        'https://www.walgreens.com/Store-Details.xml',
+    )
+
+    def parse_hours(self, days):
+        opening_hours = OpeningHours()
+        for day in days:
+            opening_hours.add_range(day=day["dayOfWeek"][:2],
+                                    open_time=day["opens"],
+                                    close_time=day["closes"])
+
+        return opening_hours.as_opening_hours()
+
+    def parse_store(self, response):
+
+        json_data = json.loads(response.xpath('//script[@id="jsonLD"]/text()').extract_first())
+
+        try:
+            street_address = json_data["address"]["streetAddress"]
+        except KeyError:
+            return  # invalid store
+
+        store_id = re.search(r'id=(.*)', response.url).groups()[0]
+
+        store_name = response.xpath('//div[contains(@class, "wag-storedetails-info")]/p/text()').extract()
+        store_name = "".join(store_name)
+        store_name = store_name.replace(u'\xa0', u' ')
+
+        props = {
+            'name': store_name,
+            'addr_full': street_address,
+            'city': json_data["address"]["addressLocality"],
+            'state': json_data["address"]["addressRegion"],
+            'postcode': json_data["address"]["postalCode"],
+            'country': json_data["address"]["addressCountry"],
+            'phone': json_data["telephone"],
+            'ref': store_id,
+            'website': response.url,
+            'opening_hours': self.parse_hours(json_data["openingHoursSpecification"]),
+            'lat': float(json_data["geo"]["latitude"]),
+            'lon': float(json_data["geo"]["longitude"]),
+            'extras': {
+                'brand': json_data.get("name") or ('Duane Reade' if 'duane' in response.url else 'Walgreens')
+            }
+        }
+        return GeojsonPointItem(**props)
+
+    def parse(self, response):
+        xml = Selector(response)
+        xml.remove_namespaces()
+
+        urls = xml.xpath('//loc/text()').extract()
+
+        for url in urls:
+            yield scrapy.Request(
+                response.urljoin(url),
+                callback=self.parse_store
+            )


Test patch: