Issue 1:
Description

I wrote my custom download middleware, which makes the request to my service(it lives on a different domain than a requested domain). After I receive a response from my service I replace URL in response.url and response.request.url to the original URL. But the issue is in parse function the URL in the response.request object is the URL from request to my service.

Steps to Reproduce

Original URL - http://example.com/some_page

URL of the cache request - http://my_service.com/?url=http://example.com/some_page

I replace the URL in middleware for request and response from step 2 to step 1

In parse function response.url is correct but request.url is from step 2

Expected behavior: request URL will be correct

Actual behavior: request URL isn't correct

Reproduces how often: each time

Versions

1.8.0 - 2.1.0

Additional context

Here is the code of my middleware

class MyServiceDownloaderMiddleware:

def process_request(self, request, spider):
if request.meta.get('__service_search') or request.meta.get('__service_action'):
return None
elif request.url.startswith(spider.eligible_url):
qs = {
'url': request.url,
}
request.meta['__service_search'] = True
request.meta['__original_url'] = request.url

new_request = request.replace(url=f'https://my_service.com/?{urlencode(qs)}')
return new_request
return None

def process_response(self, request, response, spider):
if request.meta.get('__service_action'):
request = request.replace(url=request.meta['__original_url'])
response = response.replace(url=request.meta['__original_url'], request=request)
return response
elif request.meta.get('__service_search'):
b_attribute = response.css('div.b_attr:attr(u)').extract_first()
if not b_attribute:
request = request.replace(url=request.meta['__original_url'])
request.meta.pop('__original_url')
request.meta.pop('__service_search')
response = response.replace(status=404, url=request.meta['__original_url'], request=request)

return response
_, _, d, w = b_attribute.split('|')
qs = {
'q': request.meta['__original_url'],
'd': d,
'w': w,
}
url = f'https://my_service.com/action/?{urlencode(qs)}'
new_request = request.replace(url=url)
new_request.meta['__service_action'] = True
return new_request
return response

Issue 2:
Summary

Currently requests are bound to responses in the engine and/or in the scraper.

I would like to start a conversation to figure out the implications of binding them only if the response.request attribute is not already set, i.e. something like this:

diff --git scrapy/core/engine.py scrapy/core/engine.py
index 77d71846..72f1628a 100644
--- scrapy/core/engine.py
+++ scrapy/core/engine.py
@@ -244,7 +244,8 @@ class ExecutionEngine:
% (type(response), response)
)
if isinstance(response, Response):
- response.request = request # tie request to response received
+ if response.request is None:
+ response.request = request
logkws = self.logformatter.crawled(request, response, spider)
if logkws is not None:
logger.log(*logformatter_adapter(logkws), extra={'spider': spider})

diff --git scrapy/core/scraper.py scrapy/core/scraper.py
index edbb4dd6..256935da 100644
--- scrapy/core/scraper.py
+++ scrapy/core/scraper.py
@@ -146,7 +146,8 @@ class Scraper:
self._log_download_errors, request_result, request, spider)

def call_spider(self, result, request, spider):
- result.request = request
+ if not hasattr(result, "request") or result.request is None:
+ result.request = request
dfd = defer_result(result)
callback = request.callback or spider.parse
warn_on_generator_with_return_value(spider, callback)

Motivation

Recently I've been working on a downloader middleware to transparently redirect requests to a service which takes an URL and returns its response body along with some other metadata. The process_request method replaces the URL, and the process_response decodes the response body and creates a new response, making it look like it came directly from the target website, not from the download service. At this point, I'd like to be able to restore the original request as well, but any request I return in the Response.request attribute gets overridden.

Describe alternatives you've considered

ATM I don't see a way to modify this behaviour without altering the Scrapy core.