CVE-2021-41125 (Python)

Scrapy is a high-level web crawling and scraping framework for Python. If you use `HttpAuthMiddleware` (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the `w3lib.http.basic_auth_header` function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using `HttpAuthMiddleware`.

Weakness:
CWE-522: name: Insufficiently Protected Credentials
description: The product transmits or stores authentication credentials, but it uses an insecure method that is susceptible to unauthorized interception and/or retrieval.

Repository: https://github.com/scrapy/scrapy

Vulnerable code:
file_path: scrapy/downloadermiddlewares/httpauth.py
snippet:     def spider_opened(self, spider):
        usr = getattr(spider, 'http_user', '')
        pwd = getattr(spider, 'http_pass', '')
        if usr or pwd:
            self.auth = basic_auth_header(usr, pwd)

file_path: scrapy/downloadermiddlewares/httpauth.py
snippet:     def process_request(self, request, spider):
        auth = getattr(self, 'auth', None)
        if auth and b'Authorization' not in request.headers:
            request.headers[b'Authorization'] = auth

Fixed code:
file_path: scrapy/downloadermiddlewares/httpauth.py
snippet:     def spider_opened(self, spider):
        usr = getattr(spider, 'http_user', '')
        pwd = getattr(spider, 'http_pass', '')
        if usr or pwd:
            self.auth = basic_auth_header(usr, pwd)
            if not hasattr(spider, 'http_auth_domain'):
                warnings.warn('Using HttpAuthMiddleware without http_auth_domain is deprecated and can cause security '
                              'problems if the spider makes requests to several different domains. http_auth_domain '
                              'will be set to the domain of the first request, please set it to the correct value '
                              'explicitly.',
                              category=ScrapyDeprecationWarning)
                self.domain_unset = True
            else:
                self.domain = spider.http_auth_domain
                self.domain_unset = False

file_path: scrapy/downloadermiddlewares/httpauth.py
snippet:     def process_request(self, request, spider):
        auth = getattr(self, 'auth', None)
        if auth and b'Authorization' not in request.headers:
            domain = urlparse_cached(request).hostname
            if self.domain_unset:
                self.domain = domain
                self.domain_unset = False
            if not self.domain or url_is_from_any_domain(request.url, [self.domain]):
                request.headers[b'Authorization'] = auth