scrapy/scrapy — pull request #563 by chrisboo (2014-01-26 03:40:59+00:00)
Title: SitemapSpider: Collect additional info in response.meta (new PR)

This is a new PR to replace PR #369 which got a little messy on my side. Sorry, I am still a total beginner when it comes to contributing on github! So, what is this PR about:

I found myself often scraping data from a page which was actually readily available in the sitemap which I used to get to the page. Hence, I thought it would be nice to have that sort of information in the response which is parsed in my spider. Here a little example of what this patch does:
snippet from arstechnica sitemap:

```

http://arstechnica.com/gadgets/2013/08/microsoftgoogle-bring-back-the-good-youtube-windows-phone-app/

Ars Technica
en

2013-08-14T01:00:40+00:00
Microsoft/Google bring back the good YouTube Windows Phone app
Gear & Gadgets

2013-08-14T16:01:19+00:00
daily
1.0

```

and the response.meta which make it into the spider looks like this:

```

{'priority': '1.0',
'loc': 'http://arstechnica.com/gadgets/2013/08/microsoftgoogle-bring-back-the-good-youtube-windows-phone-app/',
'lastmod': '2013-08-14T16:01:19+00:00',
'changefreq': 'daily',
'news:news': {'news:publication':
{'news:name': 'Ars Technica',
'news:language': 'en'},
'news:keywords': 'Gear & Gadgets',
'news:publication_date':
'2013-08-14T01:00:40+00:00',
'news:title': 'Microsoft/Google bring back the good YouTube Windows Phone app'}}
```

So, the improvement to the old PR is that now, all the data and hierarchy of the xml is preserved (at least for this example, more test cases from the wild would be nice)

In the old pull request there was some discussion going on whether one should put this functionality in its own spider instead of the SitemapSpider, but I believe that the changes are very small and un-intrusive now, so there might be no harm adding this to the original SitemapSpider

What do you think?