scrapy/scrapy — Issue #50: Offsite middleware ignoring port
Author: eedeep
Created: 2011-11-16 05:20:33+00:00
State: open

In my spider I have the following:

class MySpider(BaseSpider):

```python
allowed_domains = ['192.169.0.15:8080']
```

and in the parse method I do something like:

```python
yield Request('http://192.169.0.15:8080/mypage.html', self.my_callback_function)
```

the result when I run the code is that that scrapy reports:

DEBUG: Filtered offsite request to '192.168.0.15':

Which is wrong - it seems to be ignoring the port. If I change the allowed_domains to:

```python
allowed_domains = ['192.169.0.15:8080', '192.16.0.15']
```

Then it works as you would expect it to. No big deal, can work around it but I think it is a bug. The problem being located in the should_follow method of the OffsiteMiddleware class in contrib/spidermiddleware/offsite.py