Issue 1:
Hello,

I run scrapy v1.1.0

I need some parts of the url to create some ids. Lately I have been noticing that response.request.url is returning a different url that the one that is used to scrape.

In my code I use the item['LINK'] = response.request.url but that is putting in the wrong link on some scrapers like 2/10 approximately.

Is there a better way to fetch the url? The one that is displayed in the debug scrapy output is correct.

Thank you.

This one outputted in the debug window is correct:

2016-07-12 13:25:34 [scrapy] DEBUG: Scraped from <200 XXX >

The one I fetch with response.request.url is different and wrong.

Issue 2:
Description

After running the scrapy shell with the defined url, I am getting the attribute error showing the following error: AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'

scrapy shell "https://quotes.toscrape.com/tag/humor/"

Steps to Reproduce

[First Step]

[Second Step]

[and so on...]

Expected behavior: [What you expect to happen]

Actual behavior:

[2022-09-27 01:38:38 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 2.0.1, Twisted 22.8.0, Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.5 5 Jul 2022), cryptography 38.0.1, Platform Windows-10-10.0.19041-SP0

2022-09-27 01:38:38 [scrapy.crawler] INFO: Overridden settings:

{'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',

'LOGSTATS_INTERVAL': 0}

2022-09-27 01:38:38 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor

2022-09-27 01:38:38 [scrapy.extensions.telnet] INFO: Telnet Password: d760ab2d8573ec57

2022-09-27 01:38:38 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

'scrapy.extensions.telnet.TelnetConsole']

2022-09-27 01:38:39 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http"

Traceback (most recent call last):

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers_init_.py", line 49, in load_handler

dhcls = load_object(path)

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object

mod = import_module(module)

File "C:\Users\DELL\AppData\Local\Programs\Python\Python39\lib\importlib_init.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "", line 1030, in _gcd_import

File "", line 1007, in _find_and_load

File "", line 986, in _find_and_load_unlocked

File "", line 680, in _load_unlocked

File "", line 790, in exec_module

File "", line 228, in _call_with_frames_removed

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in

from scrapy.core.downloader.handlers.http11 import (

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in

from scrapy.core.downloader.contextfactory import load_context_factory_from_settings

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in

from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions

File "d:\python\scraping-project\scrapy\venv\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in

METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)

AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD']

Reproduces how often: [100%]

Versions

Scrapy : 2.6.2

lxml : 4.9.1.0

libxml2 : 2.9.12

cssselect : 1.1.0

parsel : 1.6.0

w3lib : 2.0.1

Twisted : 22.8.0

Python : 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]

pyOpenSSL : 22.1.0 (OpenSSL 3.0.5 5 Jul 2022)

cryptography : 37.0.4

Platform : Windows-10-10.0.19041-SP0

Additional context

Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use GitHub Flavored Markdown to make the text look better.