Repository: alltheplaces/alltheplaces

Issue:
Starwood Hotels
HTML hotel directory: http://www.starwoodhotels.com/corporate/directory/hotels/all/list.html?language=en_US



Discussion:
That page has links to each country, then links to each state/province/thing, then the list of hotels there.

It may be easier to extract the whole list from the map page: https://www.starwoodhotels.com/preferredguest/directory/hotels/all/map.html

`ul#resultsPrimary` has a `li` for each hotel. I'm not sure if it has *all* the info you want though. Also note that map.html redirects to a mobile page if it doesn't think your user-agent is a desktop.
When I requested data by scraper, it redirected by site to mobile version at another domain
https://assistive.usablenet.com/tt/www.starwoodhotels.com/preferredguest/directory/hotels/all/map.html?sortType=region&un_jtt_redirect
There are all data on one page
Can I use these data? or I should scrype it from genuine site (will forbid redirection)


As I said in my comment, you need to change the user agent. After some more testing: if I send a desktop web browser user agent (such as mine `Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0`) or a user agent containing the word 'bot' (`AllThePlacesBot/1.0` works!), then you don't get the redirect.

I'll let you figure out how to change the user agent in scrapy 😀
I understood it at once.
But I found simplest way to get information. 
Just want to ask is it possible to get information by that way.
If not, I'll do as it necessary
Ok

Resolving patch:
diff --git a/locations/spiders/starwoodhotels.py b/locations/spiders/starwoodhotels.py
new file mode 100644
--- /dev/null
+++ b/locations/spiders/starwoodhotels.py
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+import scrapy
+
+from locations.items import GeojsonPointItem
+
+
+class StarwoodHotelsSpider(scrapy.Spider):
+    name = "starwoodhotels"
+    allowed_domains = ["starwoodhotels.com","usablenet.com"]
+    start_urls = (
+        'https://www.starwoodhotels.com/preferredguest/directory/hotels/all/list.html?language=en_US',
+    )
+    custom_settings = {
+        'USER_AGENT': 'AllThePlacesBot/1.0',
+    }
+
+    def parse(self, response):
+        countries = response.xpath('//h5/a/@href')
+        for country in countries:
+            yield scrapy.Request(
+                response.urljoin(country.extract()),
+                callback=self.parse_country
+            )
+
+    def parse_country(self, response):
+        cities = response.xpath('//h5/a/@href')
+        for city in cities:
+            yield scrapy.Request(
+                response.urljoin(city.extract()),
+                callback=self.parse_city
+            )
+
+    def parse_city(self, response):
+        hotels = response.xpath('//a[@class="propertyName"]/@href')
+        for hotel in hotels:
+            yield scrapy.Request(
+                response.urljoin(hotel.extract()),
+                callback=self.parse_hotel
+            )
+
+    def parse_hotel(self, response):
+            props = {}  # some hotels don't have data because they don't opened yet
+
+            phone = response.xpath('//li[@class="phone"]/span/text()')
+            if phone:
+                props['phone'] = phone.extract_first().replace('Phone:', '').strip()
+
+            addr_full = response.xpath('//li[@class="street-address"]/span/text()')
+            if addr_full:
+                props['addr_full'] = addr_full.extract_first().strip()
+
+            postcode = response.xpath('//li[@class="postal-code"]/span/text()')
+            if postcode:
+                props['postcode'] = postcode.extract_first().strip()
+
+            city = response.xpath('//li[@class="city"]/span/text()')
+            if city:
+                props['city'] = city.extract_first().strip()
+
+            state = response.xpath('//li[@class="region"]/span/text()')
+            if state:
+                props['state'] = state.extract_first().strip()
+
+            country = response.xpath('//li[@class="country-name"]/span/text()')
+            if country:
+                props['country'] = country.extract_first().strip()
+
+            props['lat'] = float(response.xpath('//meta[@property="og:latitude"]/@content').extract_first())
+            props['lon'] = float(response.xpath('//meta[@property="og:longitude"]/@content').extract_first())
+            props['website'] = response.url
+            props['ref'] = response.xpath('//meta[@property="og:title"]/@content').extract_first()
+
+            yield GeojsonPointItem(**props)


Test patch: