scrapy/scrapy — pull request #369 by chrisboo (2013-08-15 21:54:45+00:00)
Title: Include additional info from sitemap into the corresponding respons' meta object

I found myself often scraping data from a page which was actually readily available in the sitemap which I used to get to the page. Hence, I thought it would be nice to have that sort of information in the response which is parsed in my spider. Here a little example of what this patch does:
snippet from arstechnica sitemap:

http://arstechnica.com/gadgets/2013/08/microsoftgoogle-bring-back-the-good-youtube-windows-phone-app/

Ars Technica
en

2013-08-14T01:00:40+00:00
Microsoft/Google bring back the good YouTube Windows Phone app
Gear & Gadgets

2013-08-14T16:01:19+00:00
daily
1.0

and the response.meta which make it into the spider looks like this:
{'loc': 'http://arstechnica.com/gadgets/2013/08/microsoftgoogle-bring-back-the-good-youtube-windows-phone-app/', 'download_latency': 0.14206790924072266, 'publication': '', 'language': 'en', 'title': 'Microsoft/Google bring back the good YouTube Windows Phone app', 'download_timeout': 180, 'priority': '1.0', 'depth': 2, 'publication_date': '2013-08-14T01:00:40+00:00', 'keywords': 'Gear & Gadgets', 'news': '', 'lastmod': '2013-08-14T16:01:19+00:00', 'changefreq': 'daily', 'download_slot': 'arstechnica.com', 'name': 'Ars Technica'}

As you can see, the hierarchic from the xml document is not preserved but I figured that a sitemap has a strict enough and short enough grammar so that this should not pose any problems.

What do you think?