Repository: scrapy/scrapy

Issue:
Offsite middleware ignoring port
In my spider I have the following:

class MySpider(BaseSpider):

``` python
    allowed_domains = ['192.169.0.15:8080']
```

and in the parse method I do something like:

``` python
    yield Request('http://192.169.0.15:8080/mypage.html', self.my_callback_function)
```

the result when I run the code is that that scrapy reports:

 DEBUG: Filtered offsite request to '192.168.0.15': <GET http://192.168.0.15:8080/mypage.html>

Which is wrong - it seems to be ignoring the port. If I change the allowed_domains to:

``` python
    allowed_domains = ['192.169.0.15:8080', '192.16.0.15']
```

Then it works as you would expect it to. No big deal, can work around it but I think it is a bug. The problem being located in the should_follow method of the OffsiteMiddleware class in contrib/spidermiddleware/offsite.py

Fixed "Offsite middleware ignoring port #50" issue
ref: https://github.com/scrapy/scrapy/issues/50

Although a [patch](https://github.com/scrapy/scrapy/issues/50#issuecomment-289200833) was suggested, simply applying it fails another test. As shown here. 

```

============================================================ FAILURES ============================================================
____________________________________________________ EngineTest.test_crawler _____________________________________________________

result = None, g = <generator object test_crawler at 0x7f396bc25780>, deferred = <Deferred at 0x7f396b6a6a28 current result: None>

    def _inlineCallbacks(result, g, deferred):
        """
        See L{inlineCallbacks}.
        """
        # This function is complicated by the need to prevent unbounded recursion
        # arising from repeatedly yielding immediately ready deferreds.  This while
        # loop and the waiting variable solve that by manually unfolding the
        # recursion.

        waiting = [True, # waiting for result?
                   None] # result

        while 1:
            try:
                # Send the last result back as the result of the yield expression.
                isFailure = isinstance(result, failure.Failure)
                if isFailure:
                    result = result.throwExceptionIntoGenerator(g)
                else:
>                   result = g.send(result)

/scrapy/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py:1386:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/scrapy/tests/test_engine.py:167: in test_crawler
    self._assert_visited_urls()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_engine.EngineTest testMethod=test_crawler>

    def _assert_visited_urls(self):
        must_be_visited = ["/", "/redirect", "/redirected",
                           "/item1.html", "/item2.html", "/item999.html"]
        urls_visited = set([rp[0].url for rp in self.run.respplug])
        urls_expected = set([self.run.geturl(p) for p in must_be_visited])
>       assert urls_expected <= urls_visited, "URLs not visited: %s" % list(urls_expected - urls_visited)
E       AssertionError: URLs not visited: ['http://localhost:42519/item2.html', 'http://localhost:42519/item999.html', 'http://localhost:42519/item1.html']

/scrapy/tests/test_engine.py:183: AssertionError
=============================================== 1 failed, 3 passed in 1.37 seconds ===============================================
ERROR: InvocationError: '/scrapy/.tox/py27/bin/py.test --cov=scrapy --cov-report= tests/test_engine.py'
____________________________________________________________ summary _____________________________________________________________

```

So, "/scrapy/tests/test_engine.py" is failing.
It looks the the test creates spiders by using  

```
allowed_domains = ["scrapytest.org", "localhost"] 
```

A spider fails to visit some urls with port number like 'http://localhost:37089/item1.html'.
Since test server is launched each time we run the test, it has different port number for local host each time. This means we cannot add a fixed port number to "localhost" in 'allowed_domain" list.

So I modified the patch to... 
```py
# scrapy/spidermiddlewares/offsite.py

    def should_follow(self, request, spider):
        regex = self.host_regex
        # hostname can be None for wrong urls (like javascript links)
        hostname = urlparse_cached(request).hostname or ''
        netloc = urlparse_cached(request).netloc or ''
        return bool(regex.search(hostname)) or bool(regex.search(netloc))
```





Discussion:
@redapple can you reopen this? The problem seems to have resurfaced recently, it happens in 1.0.3 and i didn't see any changes touching this in recent releases.
@immerrr , do you have a reproducible example?
something like this should do:

```python
# -*- coding: utf-8 -*-
import scrapy


class Localhost8000Spider(scrapy.Spider):
    name = "localhost_8000"
    allowed_domains = ["localhost:8000"]
    start_urls = (
        'http://localhost:8000/',
    )

    def parse(self, response):
        yield scrapy.Request(response.url + 'foobar')
```

Results in:
```
2016-12-07 12:23:54 [scrapy] DEBUG: Crawled (200) <GET http://localhost:8000/> (referer: None)
2016-12-07 12:23:54 [scrapy] DEBUG: Filtered offsite request to 'localhost': <GET http://localhost:8000/foobar>
```

@redapple @immerrr  I made the following change, and then it could work:

```python
# scrapy/spidermiddlewares/offsite.py

def should_follow(self, request, spider):
    regex = self.host_regex
    # pre: host = urlparse_cached(request).hostname or ''
    host = urlparse_cached(request).netloc or ''
    return bool(regex.search(host))
```
Marking this as "Good First Issue", as there's a suggested patch already. For this to be "solved", a PR should include some additional test cases for this issue.
Thank you, that fix worked for me, too. 
@cathalgarvey 
I'm trying to make a pull request (first pull request for a big project!!) now and found that applying [the patch](https://github.com/scrapy/scrapy/issues/50#issuecomment-289200833) creates another error. 
I think I need to research further...
Here's the test log.

```

============================================================ FAILURES ============================================================
____________________________________________________ EngineTest.test_crawler _____________________________________________________

result = None, g = <generator object test_crawler at 0x7f396bc25780>, deferred = <Deferred at 0x7f396b6a6a28 current result: None>

    def _inlineCallbacks(result, g, deferred):
        """
        See L{inlineCallbacks}.
        """
        # This function is complicated by the need to prevent unbounded recursion
        # arising from repeatedly yielding immediately ready deferreds.  This while
        # loop and the waiting variable solve that by manually unfolding the
        # recursion.

        waiting = [True, # waiting for result?
                   None] # result

        while 1:
            try:
                # Send the last result back as the result of the yield expression.
                isFailure = isinstance(result, failure.Failure)
                if isFailure:
                    result = result.throwExceptionIntoGenerator(g)
                else:
>                   result = g.send(result)

/scrapy/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py:1386:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/scrapy/tests/test_engine.py:167: in test_crawler
    self._assert_visited_urls()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_engine.EngineTest testMethod=test_crawler>

    def _assert_visited_urls(self):
        must_be_visited = ["/", "/redirect", "/redirected",
                           "/item1.html", "/item2.html", "/item999.html"]
        urls_visited = set([rp[0].url for rp in self.run.respplug])
        urls_expected = set([self.run.geturl(p) for p in must_be_visited])
>       assert urls_expected <= urls_visited, "URLs not visited: %s" % list(urls_expected - urls_visited)
E       AssertionError: URLs not visited: ['http://localhost:42519/item2.html', 'http://localhost:42519/item999.html', 'http://localhost:42519/item1.html']

/scrapy/tests/test_engine.py:183: AssertionError
=============================================== 1 failed, 3 passed in 1.37 seconds ===============================================
ERROR: InvocationError: '/scrapy/.tox/py27/bin/py.test --cov=scrapy --cov-report= tests/test_engine.py'
____________________________________________________________ summary _____________________________________________________________

```
"/scrapy/tests/test_engine.py" is failing.
It looks the the test creates spiders by using  

```
allowed_domains = ["scrapytest.org", "localhost"] 
```

If I apply the suggested patch,

```py
# scrapy/spidermiddlewares/offsite.py

def should_follow(self, request, spider):
    regex = self.host_regex
    # pre: host = urlparse_cached(request).hostname or ''
    host = urlparse_cached(request).netloc or ''
    return bool(regex.search(host))
```

it fails to visit some urls with port number like 'http://localhost:37089/item1.html'.
Since test server is launched each time we run the test, it has different port number for local host each time. This means we cannot add fixed a port number to "localhost" in 'allowed_domain" list.

My idea is to patch like this.
```py
# scrapy/spidermiddlewares/offsite.py

    def should_follow(self, request, spider):
        regex = self.host_regex
        # hostname can be None for wrong urls (like javascript links)
        hostname = urlparse_cached(request).hostname or ''
        netloc = urlparse_cached(request).netloc or ''
        return bool(regex.search(hostname)) or bool(regex.search(netloc))
```

Any ideas?
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/3198?src=pr&el=h1) Report
> Merging [#3198](https://codecov.io/gh/scrapy/scrapy/pull/3198?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/scrapy/commit/6c3970e6722191b642fd99c6c1bfed0d93010cab?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master    #3198      +/-   ##
==========================================
+ Coverage   82.12%   82.12%   +<.01%     
==========================================
  Files         228      228              
  Lines        9593     9594       +1     
  Branches     1385     1385              
==========================================
+ Hits         7878     7879       +1     
  Misses       1456     1456              
  Partials      259      259
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/3198?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scrapy/spidermiddlewares/offsite.py](https://codecov.io/gh/scrapy/scrapy/pull/3198/diff?src=pr&el=tree#diff-c2NyYXB5L3NwaWRlcm1pZGRsZXdhcmVzL29mZnNpdGUucHk=) | `100% <100%> (ø)` | :arrow_up: |

Guys, is there any possibility to have this patch merged? Seems brings problems with ports, apparently.

Resolving patch:
diff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.py
--- a/scrapy/spidermiddlewares/offsite.py
+++ b/scrapy/spidermiddlewares/offsite.py
@@ -53,13 +53,22 @@ def get_host_regex(self, spider):
         allowed_domains = getattr(spider, 'allowed_domains', None)
         if not allowed_domains:
             return re.compile('')  # allow all by default
-        url_pattern = re.compile("^https?://.*$")
+        url_pattern = re.compile(r"^https?://.*$")
+        port_pattern = re.compile(r":\d+$")
+        domains = []
         for domain in allowed_domains:
-            if url_pattern.match(domain):
+            if domain is None:
+                continue
+            elif url_pattern.match(domain):
                 message = ("allowed_domains accepts only domains, not URLs. "
                            "Ignoring URL entry %s in allowed_domains." % domain)
                 warnings.warn(message, URLWarning)
-        domains = [re.escape(d) for d in allowed_domains if d is not None]
+            elif port_pattern.search(domain):
+                message = ("allowed_domains accepts only domains without ports. "
+                           "Ignoring entry %s in allowed_domains." % domain)
+                warnings.warn(message, PortWarning)
+            else:
+                domains.append(re.escape(domain))
         regex = r'^(.*\.)?(%s)$' % '|'.join(domains)
         return re.compile(regex)
 
@@ -70,3 +79,7 @@ def spider_opened(self, spider):
 
 class URLWarning(Warning):
     pass
+
+
+class PortWarning(Warning):
+    pass


Test patch:
diff --git a/tests/test_spidermiddleware_offsite.py b/tests/test_spidermiddleware_offsite.py
--- a/tests/test_spidermiddleware_offsite.py
+++ b/tests/test_spidermiddleware_offsite.py
@@ -4,7 +4,7 @@
 
 from scrapy.http import Response, Request
 from scrapy.spiders import Spider
-from scrapy.spidermiddlewares.offsite import OffsiteMiddleware, URLWarning
+from scrapy.spidermiddlewares.offsite import OffsiteMiddleware, URLWarning, PortWarning
 from scrapy.utils.test import get_crawler
 
 
@@ -26,7 +26,8 @@ def test_process_spider_output(self):
                        Request('http://scrapy.org/1'),
                        Request('http://sub.scrapy.org/1'),
                        Request('http://offsite.tld/letmepass', dont_filter=True),
-                       Request('http://scrapy.test.org/')]
+                       Request('http://scrapy.test.org/'),
+                       Request('http://scrapy.test.org:8000/')]
         offsite_reqs = [Request('http://scrapy2.org'),
                        Request('http://offsite.tld/'),
                        Request('http://offsite.tld/scrapytest.org'),
@@ -55,21 +56,21 @@ def test_process_spider_output(self):
 
 class TestOffsiteMiddleware3(TestOffsiteMiddleware2):
 
-    def _get_spider(self):
-        return Spider('foo')
+    def _get_spiderargs(self):
+        return dict(name='foo')
 
 
 class TestOffsiteMiddleware4(TestOffsiteMiddleware3):
 
-    def _get_spider(self):
-      bad_hostname = urlparse('http:////scrapytest.org').hostname
-      return dict(name='foo', allowed_domains=['scrapytest.org', None, bad_hostname])
+    def _get_spiderargs(self):
+        bad_hostname = urlparse('http:////scrapytest.org').hostname
+        return dict(name='foo', allowed_domains=['scrapytest.org', None, bad_hostname])
 
     def test_process_spider_output(self):
-      res = Response('http://scrapytest.org')
-      reqs = [Request('http://scrapytest.org/1')]
-      out = list(self.mw.process_spider_output(res, reqs, self.spider))
-      self.assertEqual(out, reqs)
+        res = Response('http://scrapytest.org')
+        reqs = [Request('http://scrapytest.org/1')]
+        out = list(self.mw.process_spider_output(res, reqs, self.spider))
+        self.assertEqual(out, reqs)
 
 
 class TestOffsiteMiddleware5(TestOffsiteMiddleware4):
@@ -80,3 +81,13 @@ def test_get_host_regex(self):
             warnings.simplefilter("always")
             self.mw.get_host_regex(self.spider)
             assert issubclass(w[-1].category, URLWarning)
+
+
+class TestOffsiteMiddleware6(TestOffsiteMiddleware4):
+
+    def test_get_host_regex(self):
+        self.spider.allowed_domains = ['scrapytest.org:8000', 'scrapy.org', 'scrapy.test.org']
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+            self.mw.get_host_regex(self.spider)
+            assert issubclass(w[-1].category, PortWarning)