scrapy/scrapy — pull request #152 by mouadino (2012-06-29 19:14:59+00:00)
Title: Make sure that the response's meta is not discarded when passed to the SpiderCrawl._requests_to_follow

**Use Case:**

One of my spiders have the the following:

```python
class MySpider(CrawlSpider):
...
def start_requests(self):
...
for i in ...
yield FormRequest(url, method='POST',
formdata={'page': i},
callback=self._requests_to_follow,
meta={...}
)
```

As you can see, i am creating a new ``Response`` while passing it some data as meta to be able to retrieve it in my main parse method.

Without the following patch, the ``_requests_to_follow`` method was creating a new ``Response`` but this latest didn't have the meta that i passed, which IMHO i consider a bug.

**My fix** was to extend the ``_requests_to_follow`` method as bellow but i am hopping that this bug can be fixed in scrapy directly:

```python
def _requests_to_follow(self, response):
"""Extend the parent ``_requests_to_follow`` method to be able to pass
the original response's meta data that the original method cut off when
creating response from the rules.

"""
for new_response in \
super(MySpider, self)._requests_to_follow(response):
new_response.meta.update(response.meta)
yield new_response
```

N.B: I didn't write any unit test for this b/c i didn't find a simple way to test it, FYI i did run all the tests after the change to make sure i didn't break anything :), so if the tests are required please advise me as to the best way to do it.

Thanks,