Repository: platformsh/platformsh-docs

Issue:
Bump scrapy from 2.6.1 to 2.6.2 in /search
Bumps [scrapy](https://github.com/scrapy/scrapy) from 2.6.1 to 2.6.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/scrapy/scrapy/releases">scrapy's releases</a>.</em></p>
<blockquote>
<h2>2.6.2</h2>
<p>Fixes a <strong>security issue</strong> around HTTP proxy usage, and addresses a few regressions introduced in Scrapy 2.6.0.</p>
<p>See the <strong><a href="https://docs.scrapy.org/en/2.6/news.html#scrapy-2-6-2-2022-07-25">changelog</a></strong>.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/scrapy/scrapy/blob/master/docs/news.rst">scrapy's changelog</a>.</em></p>
<blockquote>
<h2>Scrapy 2.6.2 (2022-07-25)</h2>
<p><strong>Security bug fix:</strong></p>
<ul>
<li>
<p>When :class:<code>~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware</code>
processes a request with :reqmeta:<code>proxy</code> metadata, and that
:reqmeta:<code>proxy</code> metadata includes proxy credentials,
:class:<code>~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware</code> sets
the <code>Proxy-Authentication</code> header, but only if that header is not already
set.</p>
<p>There are third-party proxy-rotation downloader middlewares that set
different :reqmeta:<code>proxy</code> metadata every time they process a request.</p>
<p>Because of request retries and redirects, the same request can be processed
by downloader middlewares more than once, including both
:class:<code>~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware</code> and
any third-party proxy-rotation downloader middleware.</p>
<p>These third-party proxy-rotation downloader middlewares could change the
:reqmeta:<code>proxy</code> metadata of a request to a new value, but fail to remove
the <code>Proxy-Authentication</code> header from the previous value of the
:reqmeta:<code>proxy</code> metadata, causing the credentials of one proxy to be sent
to a different proxy.</p>
<p>To prevent the unintended leaking of proxy credentials, the behavior of
:class:<code>~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware</code> is now
as follows when processing a request:</p>
<ul>
<li>
<p>If the request being processed defines :reqmeta:<code>proxy</code> metadata that
includes credentials, the <code>Proxy-Authorization</code> header is always
updated to feature those credentials.</p>
</li>
<li>
<p>If the request being processed defines :reqmeta:<code>proxy</code> metadata
without credentials, the <code>Proxy-Authorization</code> header is removed
<em>unless</em> it was originally defined for the same proxy URL.</p>
<p>To remove proxy credentials while keeping the same proxy URL, remove
the <code>Proxy-Authorization</code> header.</p>
</li>
<li>
<p>If the request has no :reqmeta:<code>proxy</code> metadata, or that metadata is a
falsy value (e.g. <code>None</code>), the <code>Proxy-Authorization</code> header is
removed.</p>
<p>It is no longer possible to set a proxy URL through the
:reqmeta:<code>proxy</code> metadata but set the credentials through the
<code>Proxy-Authorization</code> header. Set proxy credentials through the
:reqmeta:<code>proxy</code> metadata instead.</p>
</li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/scrapy/scrapy/commit/aecbccbaa567b07694141a4503e9abf1bb2c919f"><code>aecbccb</code></a> Bump version: 2.6.1 → 2.6.2</li>
<li><a href="https://github.com/scrapy/scrapy/commit/af7dd16d8ded3e6cb2946603688f4f4a5212e80f"><code>af7dd16</code></a> Merge pull request from GHSA-9x8m-2xpf-crp3</li>
<li><a href="https://github.com/scrapy/scrapy/commit/42056090516bb0cc5d349e232298c711ec452bc5"><code>4205609</code></a> Fixed intersphinx references</li>
<li><a href="https://github.com/scrapy/scrapy/commit/e3e69d1209407c72a6478936bdbfd32cc22e9432"><code>e3e69d1</code></a> Pin documentation requirements (<a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5536">#5536</a>)</li>
<li><a href="https://github.com/scrapy/scrapy/commit/54bfb9649bdec565f9798cc41643ed1bae25bd67"><code>54bfb96</code></a> Cover <a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5525">#5525</a> in the 2.6.2 release notes (<a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5535">#5535</a>)</li>
<li><a href="https://github.com/scrapy/scrapy/commit/4ef71829b22b7362d08d6897090595138107852f"><code>4ef7182</code></a> If TWISTED_REACTOR is None, reuse any pre-installed reactor (<a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5528">#5528</a>)</li>
<li><a href="https://github.com/scrapy/scrapy/commit/1c1cd5d8eae48eade88560426499846d217a555f"><code>1c1cd5d</code></a> Update the 2.6.2 release notes</li>
<li><a href="https://github.com/scrapy/scrapy/commit/84c29a286f6b9bc94a5318ca68cd4fc21244443a"><code>84c29a2</code></a> Unset the release date of still-unreleased 2.6.2 (<a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5503">#5503</a>)</li>
<li><a href="https://github.com/scrapy/scrapy/commit/b9b9422bb1025d00c5113994490ac71a162d775f"><code>b9b9422</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/scrapy/scrapy/issues/5482">#5482</a> from alexpdev/parse_help_msg</li>
<li><a href="https://github.com/scrapy/scrapy/commit/915c288205e2b9a0bdbbe18cc67cd23ba5bb4de3"><code>915c288</code></a> edit</li>
<li>Additional commits viewable in <a href="https://github.com/scrapy/scrapy/compare/2.6.1...2.6.2">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=scrapy&package-manager=pip&previous-version=2.6.1&new-version=2.6.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/platformsh/platformsh-docs/network/alerts).

</details>


Discussion:

Resolving patch:
diff --git a/search/docs/middlewares.py b/search/docs/middlewares.py
--- a/search/docs/middlewares.py
+++ b/search/docs/middlewares.py
@@ -6,6 +6,7 @@
 # https://docs.scrapy.org/en/latest/topics/spider-middleware.html
 
 from scrapy import signals
+from scrapy.exceptions import IgnoreRequest
 
 
 class DocsSpiderMiddleware(object):
@@ -61,13 +62,6 @@ class DocsDownloaderMiddleware(object):
     # scrapy acts as if the downloader middleware does not modify the
     # passed objects.
 
-    @classmethod
-    def from_crawler(cls, crawler):
-        # This method is used by Scrapy to create your spiders.
-        s = cls()
-        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
-        return s
-
     def process_request(self, request, spider):
         # Called for each request that goes through the downloader
         # middleware.
@@ -78,26 +72,7 @@ def process_request(self, request, spider):
         # - or return a Request object
         # - or raise IgnoreRequest: process_exception() methods of
         #   installed downloader middleware will be called
-        return None
-
-    def process_response(self, request, response, spider):
-        # Called with the response returned from the downloader.
+        if request.url.endswith(".png") or request.url.endswith(".jpg"):
+            raise IgnoreRequest
 
-        # Must either;
-        # - return a Response object
-        # - return a Request object
-        # - or raise IgnoreRequest
-        return response
-
-    def process_exception(self, request, exception, spider):
-        # Called when a download handler or a process_request()
-        # (from other downloader middleware) raises an exception.
-
-        # Must either:
-        # - return None: continue processing this exception
-        # - return a Response object: stops process_exception() chain
-        # - return a Request object: stops process_exception() chain
-        pass
-
-    def spider_opened(self, spider):
-        spider.logger.info('Spider opened: %s' % spider.name)
+        return None
diff --git a/search/docs/settings.py b/search/docs/settings.py
--- a/search/docs/settings.py
+++ b/search/docs/settings.py
@@ -52,9 +52,9 @@
 
 # Enable or disable downloader middlewares
 # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
-#DOWNLOADER_MIDDLEWARES = {
-#    'docs.middlewares.DocsDownloaderMiddleware': 543,
-#}
+DOWNLOADER_MIDDLEWARES = {
+    'docs.middlewares.DocsDownloaderMiddleware': 543,
+}
 
 # Enable or disable extensions
 # See https://docs.scrapy.org/en/latest/topics/extensions.html


Test patch: