Repository: scrapy/scrapy
Pull request #3030 (2017-12-04 19:27:10)

Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999)
Add mention to dont_merge_cookies in the Documentation of CookiesMIddlewares issue: #2999 


Patch:
diff --git a/.travis.yml b/.travis.yml
index d4f30814d..6635f5d3b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -13,8 +13,10 @@ matrix:
       env: TOXENV=jessie
     - python: 2.7
       env: TOXENV=pypy
-    - python: 3.3
-      env: TOXENV=py33
+    - python: 2.7
+      env: TOXENV=pypy3
+    - python: 3.4
+      env: TOXENV=py34
     - python: 3.5
       env: TOXENV=py35
     - python: 3.6
@@ -30,6 +32,13 @@ install:
         virtualenv --python="$PYPY_VERSION/bin/pypy" "$HOME/virtualenvs/$PYPY_VERSION"
         source "$HOME/virtualenvs/$PYPY_VERSION/bin/activate"
       fi
+      if [ "$TOXENV" = "pypy3" ]; then
+        export PYPY_VERSION="pypy3.5-5.9-beta-linux_x86_64-portable"
+        wget "https://bitbucket.org/squeaky/portable-pypy/downloads/${PYPY_VERSION}.tar.bz2"
+        tar -jxf ${PYPY_VERSION}.tar.bz2
+        virtualenv --python="$PYPY_VERSION/bin/pypy3" "$HOME/virtualenvs/$PYPY_VERSION"
+        source "$HOME/virtualenvs/$PYPY_VERSION/bin/activate"
+      fi
   - pip install -U tox twine wheel codecov
 
 script: tox
diff --git a/README.rst b/README.rst
index 45135c7a2..1361eac26 100644
--- a/README.rst
+++ b/README.rst
@@ -6,6 +6,10 @@ Scrapy
    :target: https://pypi.python.org/pypi/Scrapy
    :alt: PyPI Version
 
+.. image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
+   :target: https://pypi.python.org/pypi/Scrapy
+   :alt: Supported Python Versions
+
 .. image:: https://img.shields.io/travis/scrapy/scrapy/master.svg
    :target: https://travis-ci.org/scrapy/scrapy
    :alt: Build Status
@@ -36,7 +40,7 @@ https://scrapy.org
 Requirements
 ============
 
-* Python 2.7 or Python 3.3+
+* Python 2.7 or Python 3.4+
 * Works on Linux, Windows, Mac OSX, BSD
 
 Install
diff --git a/docs/contributing.rst b/docs/contributing.rst
index f3732ab06..9a02634cb 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -19,12 +19,16 @@ There are many ways to contribute to Scrapy. Here are some of them:
   the guidelines detailed in `Reporting bugs`_ below.
 
 * Submit patches for new functionality and/or bug fixes. Please read
-  `Writing patches`_ and `Submitting patches`_ below for details on how to
+  :ref:`writing-patches` and `Submitting patches`_ below for details on how to
   write and submit a patch.
 
 * Join the `Scrapy subreddit`_ and share your ideas on how to
   improve Scrapy. We're always open to suggestions.
 
+* Answer Scrapy questions at
+  `Stack Overflow <https://stackoverflow.com/questions/tagged/scrapy>`__.
+
+
 Reporting bugs
 ==============
 
@@ -40,9 +44,14 @@ guidelines when reporting a new bug.
 * check the :ref:`FAQ <faq>` first to see if your issue is addressed in a
   well-known question
 
+* if you have a general question about scrapy usage, please ask it at
+  `Stack Overflow <https://stackoverflow.com/questions/tagged/scrapy>`__
+  (use "scrapy" tag).
+
 * check the `open issues`_ to see if it has already been reported. If it has,
-  don't dismiss the report but check the ticket history and comments, you may
-  find additional useful information to contribute.
+  don't dismiss the report, but check the ticket history and comments. If you
+  have additional useful information, please leave a comment, or consider
+  :ref:`sending a pull request <writing-patches>` with a fix.
 
 * search the `scrapy-users`_ list and `Scrapy subreddit`_ to see if it has
   been discussed there, or if you're not sure if what you're seeing is a bug.
@@ -54,12 +63,20 @@ guidelines when reporting a new bug.
   it. See for example StackOverflow's guide on creating a
   `Minimal, Complete, and Verifiable example`_ exhibiting the issue.
 
+* the most awesome way to provide a complete reproducible example is to
+  send a pull request which adds a failing test case to the
+  Scrapy testing suite (see :ref:`submitting-patches`).
+  This is helpful even if you don't have an intention to
+  fix the issue yourselves.
+
 * include the output of ``scrapy version -v`` so developers working on your bug
   know exactly which version and platform it occurred on, which is often very
   helpful for reproducing it, or knowing if it was already fixed.
 
 .. _Minimal, Complete, and Verifiable example: https://stackoverflow.com/help/mcve
 
+.. _writing-patches:
+
 Writing patches
 ===============
 
@@ -83,6 +100,8 @@ Well-written patches should:
   the documentation changes in the same patch.  See `Documentation policies`_
   below.
 
+.. _submitting-patches:
+
 Submitting patches
 ==================
 
@@ -100,11 +119,29 @@ starting point is to send a pull request on GitHub. It can be simple enough to
 illustrate your idea, and leave documentation/tests for later, after the idea
 has been validated and proven useful. Alternatively, you can start a
 conversation in the `Scrapy subreddit`_ to discuss your idea first.
+
+Sometimes there is an existing pull request for the problem you'd like to
+solve, which is stalled for some reason. Often the pull request is in a
+right direction, but changes are requested by Scrapy maintainers, and the
+original pull request author haven't had time to address them.
+In this case consider picking up this pull request: open
+a new pull request with all commits from the original pull request, as well as
+additional changes to address the raised issues. Doing so helps a lot; it is
+not considered rude as soon as the original author is acknowledged by keeping
+his/her commits.
+
+You can pull an existing pull request to a local branch
+by running ``git fetch upstream pull/$PR_NUMBER/head:$BRANCH_NAME_TO_CREATE``
+(replace 'upstream' with a remote name for scrapy repository,
+``$PR_NUMBER`` with an ID of the pull request, and ``$BRANCH_NAME_TO_CREATE``
+with a name of the branch you want to create locally).
+See also: https://help.github.com/articles/checking-out-pull-requests-locally/#modifying-an-inactive-pull-request-locally.
+
 When writing GitHub pull requests, try to keep titles short but descriptive.
 E.g. For bug #411: "Scrapy hangs if an exception raises in start_requests"
 prefer "Fix hanging when exception occurs in start_requests (#411)"
-instead of "Fix for #411".
-Complete titles make it easy to skim through the issue tracker.
+instead of "Fix for #411". Complete titles make it easy to skim through
+the issue tracker.
 
 Finally, try to keep aesthetic changes (:pep:`8` compliance, unused imports
 removal, etc) in separate commits than functional changes. This will make pull
@@ -121,21 +158,29 @@ Scrapy:
 * It's OK to use lines longer than 80 chars if it improves the code
   readability.
 
-* Don't put your name in the code you contribute. Our policy is to keep
-  the contributor's name in the `AUTHORS`_ file distributed with Scrapy.
+* Don't put your name in the code you contribute; git provides enough
+  metadata to identify author of the code.
+  See https://help.github.com/articles/setting-your-username-in-git/ for
+  setup instructions.
 
 Documentation policies
 ======================
 
 * **Don't** use docstrings for documenting classes, or methods which are
-  already documented in the official (sphinx) documentation. For example, the
-  :meth:`ItemLoader.add_value` method should be documented in the sphinx
-  documentation, not its docstring.
+  already documented in the official (sphinx) documentation. Alternatively,
+  **do** provide a docstring, but make sure sphinx documentation uses
+  autodoc_ extension to pull the docstring. For example, the
+  :meth:`ItemLoader.add_value` method should be either
+  documented only in the sphinx documentation (not it a docstring), or
+  it should have a docstring which is pulled to sphinx documentation using
+  autodoc_ extension.
 
 * **Do** use docstrings for documenting functions not present in the official
   (sphinx) documentation, such as functions from ``scrapy.utils`` package and
   its sub-modules.
 
+.. _autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html
+
 Tests
 =====
 
diff --git a/docs/faq.rst b/docs/faq.rst
index 42c3abbfa..7a0628f88 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -69,9 +69,11 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML pars
 What Python versions does Scrapy support?
 -----------------------------------------
 
-Scrapy is supported under Python 2.7 and Python 3.3+.
+Scrapy is supported under Python 2.7 and Python 3.4+
+under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
 Python 2.6 support was dropped starting at Scrapy 0.20.
 Python 3 support was added in Scrapy 1.1.
+PyPy support was added in Scrapy 1.4, PyPy3 support was added in Scrapy 1.5.
 
 .. note::
     For Python 3 support on Windows, it is recommended to use
diff --git a/docs/intro/install.rst b/docs/intro/install.rst
index 12d489612..4a9aa3cfb 100644
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@@ -7,7 +7,8 @@ Installation guide
 Installing Scrapy
 =================
 
-Scrapy runs on Python 2.7 and Python 3.3 or above.
+Scrapy runs on Python 2.7 and Python 3.4 or above
+under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
 
 If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
 the `conda-forge`_ channel, which has up-to-date packages for Linux, Windows
@@ -132,12 +133,12 @@ Once you've installed `Anaconda`_ or `Miniconda`_, install Scrapy with::
 
 .. _intro-install-ubuntu:
 
-Ubuntu 12.04 or above
+Ubuntu 14.04 or above
 ---------------------
 
 Scrapy is currently tested with recent-enough versions of lxml,
 twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions.
-But it should support older versions of Ubuntu too, like Ubuntu 12.04,
+But it should support older versions of Ubuntu too, like Ubuntu 14.04,
 albeit with potential issues with TLS connections.
 
 **Don't** use the ``python-scrapy`` package provided by Ubuntu, they are
@@ -163,8 +164,8 @@ you can install Scrapy with ``pip`` after that::
     pip install scrapy
 
 .. note::
-    The same non-python dependencies can be used to install Scrapy in Debian
-    Wheezy (7.0) and above.
+    The same non-Python dependencies can be used to install Scrapy in Debian
+    Jessie (8.0) and above.
 
 
 .. _intro-install-macos:
@@ -223,6 +224,29 @@ After any of these workarounds you should be able to install Scrapy::
   pip install Scrapy
 
 
+PyPy
+----
+
+We recommend using the latest PyPy version. The version tested is 5.9.0.
+For PyPy3, only Linux installation was tested.
+
+Most scrapy dependencides now have binary wheels for CPython, but not for PyPy.
+This means that these dependecies will be built during installation.
+On OS X, you are likely to face an issue with building Cryptography dependency,
+solution to this problem is described
+`here <https://github.com/pyca/cryptography/issues/2692#issuecomment-272773481>`_,
+that is to ``brew install openssl`` and then export the flags that this command
+recommends (only needed when installing scrapy). Installing on Linux has no special
+issues besides installing build dependencies.
+Installing scrapy with PyPy on Windows is not tested.
+
+You can check that scrapy is installed correctly by running ``scrapy bench``.
+If this command gives errors such as
+``TypeError: ... got 2 unexpected keyword arguments``, this means
+that setuptools was unable to pick up one PyPy-specific dependency.
+To fix this issue, run ``pip install 'PyPyDispatcher>=2.1.0'``.
+
+
 .. _Python: https://www.python.org/
 .. _pip: https://pip.pypa.io/en/latest/installing/
 .. _lxml: http://lxml.de/
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 983a93290..863620900 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -237,6 +237,17 @@ Default: ``True``
 Whether to enable the cookies middleware. If disabled, no cookies will be sent
 to web servers.
 
+Notice that if the :class:`~scrapy.http.Request` 
+has ``meta['dont_merge_cookies']`` evaluated to ``True``. 
+despite the value of :setting:`COOKIES_ENABLED` the cookies will **not** be 
+sent to web servers and received cookies in 
+:class:`~scrapy.http.Response` will **not** be merged with the existing 
+cookies.
+
+For more detailed information see the ``cookies`` parameter in 
+:class:`~scrapy.http.Request`
+
+
 .. setting:: COOKIES_DEBUG
 
 COOKIES_DEBUG
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
index 48d2481b4..8e9727093 100644
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -15,6 +15,10 @@ from twisted.internet.error import TimeoutError
 from twisted.web.http import _DataLoss, PotentialDataLoss
 from twisted.web.client import Agent, ProxyAgent, ResponseDone, \
     HTTPConnectionPool, ResponseFailed
+try:
+    from twisted.web.client import URI
+except ImportError:
+    from twisted.web.client import _URI as URI
 from twisted.internet.endpoints import TCP4ClientEndpoint
 
 from scrapy.http import Headers
@@ -228,10 +232,38 @@ class TunnelingAgent(Agent):
             headers, bodyProducer, requestPath)
 
 
+class ScrapyProxyAgent(Agent):
+
+    def __init__(self, reactor, proxyURI,
+                 connectTimeout=None, bindAddress=None, pool=None):
+        super(ScrapyProxyAgent, self).__init__(reactor,
+                                               connectTimeout=connectTimeout,
+                                               bindAddress=bindAddress,
+                                               pool=pool)
+        self._proxyURI = URI.fromBytes(proxyURI)
+
+    def request(self, method, uri, headers=None, bodyProducer=None):
+        """
+        Issue a new request via the configured proxy.
+        """
+        # Cache *all* connections under the same key, since we are only
+        # connecting to a single destination, the proxy:
+        if twisted_version >= (15, 0, 0):
+            proxyEndpoint = self._getEndpoint(self._proxyURI)
+        else:
+            proxyEndpoint = self._getEndpoint(self._proxyURI.scheme,
+                                              self._proxyURI.host,
+                                              self._proxyURI.port)
+        key = ("http-proxy", self._proxyURI.host, self._proxyURI.port)
+        return self._requestWithEndpoint(key, proxyEndpoint, method,
+                                         URI.fromBytes(uri), headers,
+                                         bodyProducer, uri)
+
+
 class ScrapyAgent(object):
 
     _Agent = Agent
-    _ProxyAgent = ProxyAgent
+    _ProxyAgent = ScrapyProxyAgent
     _TunnelingAgent = TunnelingAgent
 
     def __init__(self, contextFactory=None, connectTimeout=10, bindAddress=None, pool=None,
@@ -260,9 +292,8 @@ class ScrapyAgent(object):
                     contextFactory=self._contextFactory, connectTimeout=timeout,
                     bindAddress=bindaddress, pool=self._pool)
             else:
-                endpoint = TCP4ClientEndpoint(reactor, proxyHost, proxyPort,
-                    timeout=timeout, bindAddress=bindaddress)
-                return self._ProxyAgent(endpoint)
+                return self._ProxyAgent(reactor, proxyURI=to_bytes(proxy, encoding='ascii'),
+                    connectTimeout=timeout, bindAddress=bindaddress, pool=self._pool)
 
         return self._Agent(reactor, contextFactory=self._contextFactory,
             connectTimeout=timeout, bindAddress=bindaddress, pool=self._pool)
diff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.py
index ea1c9270f..310166cad 100644
--- a/scrapy/spidermiddlewares/offsite.py
+++ b/scrapy/spidermiddlewares/offsite.py
@@ -6,6 +6,7 @@ See documentation in docs/topics/spider-middleware.rst
 
 import re
 import logging
+import warnings
 
 from scrapy import signals
 from scrapy.http import Request
@@ -52,9 +53,18 @@ class OffsiteMiddleware(object):
         allowed_domains = getattr(spider, 'allowed_domains', None)
         if not allowed_domains:
             return re.compile('') # allow all by default
+        url_pattern = re.compile("^https?://.*$")
+        for domain in allowed_domains:
+            if url_pattern.match(domain):
+                warnings.warn("allowed_domains accepts only domains, not URLs. Ignoring URL entry %s in allowed_domains." % domain, URLWarning)
+                
         regex = r'^(.*\.)?(%s)$' % '|'.join(re.escape(d) for d in allowed_domains if d is not None)
         return re.compile(regex)
 
     def spider_opened(self, spider):
         self.host_regex = self.get_host_regex(spider)
         self.domains_seen = set()
+
+
+class URLWarning(Warning):
+    pass
diff --git a/setup.py b/setup.py
index 327286f5a..06a36e2ba 100644
--- a/setup.py
+++ b/setup.py
@@ -53,14 +53,16 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: Implementation :: CPython',
+        'Programming Language :: Python :: Implementation :: PyPy',
         'Topic :: Internet :: WWW/HTTP',
         'Topic :: Software Development :: Libraries :: Application Frameworks',
         'Topic :: Software Development :: Libraries :: Python Modules',
     ],
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     install_requires=[
         'Twisted>=13.1.0',
         'w3lib>=1.17.0',
diff --git a/tox.ini b/tox.ini
index f35b894f3..60ff8c15e 100644
--- a/tox.ini
+++ b/tox.ini
@@ -63,25 +63,27 @@ basepython = pypy
 commands =
     py.test {posargs:scrapy tests}
 
-[testenv:py33]
-basepython = python3.3
+[testenv:py34]
+basepython = python3.4
 deps =
     -rrequirements-py3.txt
     # Extras
     Pillow
     -rtests/requirements-py3.txt
 
-[testenv:py34]
-basepython = python3.4
-deps = {[testenv:py33]deps}
-
 [testenv:py35]
 basepython = python3.5
-deps = {[testenv:py33]deps}
+deps = {[testenv:py34]deps}
 
 [testenv:py36]
 basepython = python3.6
-deps = {[testenv:py33]deps}
+deps = {[testenv:py34]deps}
+
+[testenv:pypy3]
+basepython = pypy3
+deps = {[testenv:py34]deps}
+commands =
+    py.test {posargs:scrapy tests}
 
 [docs]
 changedir = docs


Test patch:
diff --git a/tests/test_spidermiddleware_offsite.py b/tests/test_spidermiddleware_offsite.py
index 9ad86313c..7e4af0d4c 100644
--- a/tests/test_spidermiddleware_offsite.py
+++ b/tests/test_spidermiddleware_offsite.py
@@ -5,7 +5,9 @@ from six.moves.urllib.parse import urlparse
 from scrapy.http import Response, Request
 from scrapy.spiders import Spider
 from scrapy.spidermiddlewares.offsite import OffsiteMiddleware
+from scrapy.spidermiddlewares.offsite import URLWarning
 from scrapy.utils.test import get_crawler
+import warnings
 
 class TestOffsiteMiddleware(TestCase):
 
@@ -68,3 +70,13 @@ class TestOffsiteMiddleware4(TestOffsiteMiddleware3):
       reqs = [Request('http://scrapytest.org/1')]
       out = list(self.mw.process_spider_output(res, reqs, self.spider))
       self.assertEqual(out, reqs)
+
+
+class TestOffsiteMiddleware5(TestOffsiteMiddleware4):
+    
+    def test_get_host_regex(self):
+        self.spider.allowed_domains = ['http://scrapytest.org', 'scrapy.org', 'scrapy.test.org']
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+            self.mw.get_host_regex(self.spider)
+            assert issubclass(w[-1].category, URLWarning)
diff --git a/tests/test_utils_python.py b/tests/test_utils_python.py
index 115f523e9..f6133657b 100644
--- a/tests/test_utils_python.py
+++ b/tests/test_utils_python.py
@@ -219,9 +219,12 @@ class UtilsPythonTestCase(unittest.TestCase):
             self.assertEqual(get_func_args(" ".join), [])
             self.assertEqual(get_func_args(operator.itemgetter(2)), [])
         else:
-            self.assertEqual(get_func_args(six.text_type.split), ['sep', 'maxsplit'])
-            self.assertEqual(get_func_args(" ".join), ['list'])
-            self.assertEqual(get_func_args(operator.itemgetter(2)), ['obj'])
+            stripself = not six.PY2  # PyPy3 exposes them as methods
+            self.assertEqual(
+                get_func_args(six.text_type.split, stripself), ['sep', 'maxsplit'])
+            self.assertEqual(get_func_args(" ".join, stripself), ['list'])
+            self.assertEqual(
+                get_func_args(operator.itemgetter(2), stripself), ['obj'])
 
 
     def test_without_none_values(self):