Issue 1:
if allowed_domains occurred in url(in path or query) it also crawl that url.

e.g:

allowed_domains =["www.rankwatch.com"]

when i run LinkExtractor it also crawl :

{"url": "https://www.facebook.com/login.php?skip_api_login=1&api_key=864596490301578&signed_next=1&next=https%3A%2F%2Fwww.facebook.com%2Fv2.4%2Fdialog%2Foauth%3Fredirect_uri%3Dhttps%253A%252F%252Fwww.rankwatch.com%252Flearning%252Fuser%252Fsimple-fb-connect%252Freturn%26state%3Daa68bcc193612233a0511106f69f9203%26scope%3Demail%26client_id%3D864596490301578%26ret%3Dlogin%26sdk%3Dphp-sdk-4.0.23%26logger_id%3Df7320741-707c-2f7e-1947-390b18a93e99&cancel_url=https%3A%2F%2Fwww.rankwatch.com%2Flearning%2Fuser%2Fsimple-fb-connect%2Freturn%3Ferror%3Daccess_denied%26error_code%3D200%26error_description%3DPermissions%2Berror%26error_reason%3Duser_denied%26state%3Daa68bcc193612233a0511106f69f9203%23_%3D_&display=page&locale=en_GB&logger_id=f7320741-707c-2f7e-1947-390b18a93e99"},

Issue 2:
Hello,

To avoid crawling duplicated URLs, I store all crawled URLs in database, and then check if one URL requested is already in DB or not.

Here is my filter class:

class CustomFilter(RFPDupeFilter):
"""A dupe filter that considers specific ids in the url"""
db = DBManager(settings[ 'MONGODB_VISITED_URLS' ])

def request_seen(self, request):
if self.db.exist("url", request.url):
return True
else:
visitedUrl = VisitedURL()
visitedUrl['url'] = request.url
visitedUrl['orgDupURL'] = None
self.db.insert(visitedUrl)
return False

However I found request_seen method gets called 3 times for one page crawling, here are stack trace for the 3 calls in order:

------------------------------ 1

request_seen [filter.py:18]

enqueue_request [scheduler.py:51]

schedule [engine.py:189]

crawl [engine.py:183]

_process_spidermw_output [scraper.py:183]

[defer.py:63]

_oneWorkUnit [task.py:491]

_tick [task.py:645]

runUntilCurrent [base.py:825]

mainLoop [base.py:1203]

run [base.py:1194]

start [crawler.py:251]

[start.py:15]

run [pydevd.py:931]

[pydevd.py:1524]

------------------------------ 2

request_seen [filter.py:18]

enqueue_request [scheduler.py:51]

schedule [engine.py:189]

crawl [engine.py:183]

_handle_downloader_output [engine.py:155]

_runCallbacks [defer.py:588]

_startRunCallbacks [defer.py:501]

callback [defer.py:393]

_runCallbacks [defer.py:588]

_startRunCallbacks [defer.py:501]

callback [defer.py:393]

connectionLost [http11.py:320]

_bodyDataFinished_CONNECTED [_newclient.py:1161]

dispatcher [_newclient.py:916]

connectionLost [_newclient.py:537]

_disconnectParser [_newclient.py:1513]

_finishResponse_WAITING [_newclient.py:1487]

dispatcher [_newclient.py:916]

_finished [_newclient.py:440]

dataReceived [http.py:1478]

rawDataReceived [_newclient.py:299]

dataReceived [basic.py:578]

dataReceived [_newclient.py:385]

dataReceived [_newclient.py:1533]

dataReceived [endpoints.py:102]

_dataReceived [tcp.py:215]

doRead [tcp.py:209]

_doReadOrWrite [posixbase.py:597]

callWithContext [context.py:81]

callWithContext [context.py:118]

callWithContext [log.py:84]

callWithLogger [log.py:101]

doPoll [epollreactor.py:396]

mainLoop [base.py:1206]

run [base.py:1194]

start [crawler.py:251]

[start.py:15]

run [pydevd.py:931]

[pydevd.py:1524]

------------------------------ 3

request_seen [filter.py:18]

enqueue_request [scheduler.py:51]

schedule [engine.py:189]

crawl [engine.py:183]

_handle_downloader_output [engine.py:155]

_runCallbacks [defer.py:588]

_startRunCallbacks [defer.py:501]

callback [defer.py:393]

_runCallbacks [defer.py:588]

_startRunCallbacks [defer.py:501]

callback [defer.py:393]

connectionLost [http11.py:320]

_bodyDataFinished_CONNECTED [_newclient.py:1161]

dispatcher [_newclient.py:916]

connectionLost [_newclient.py:537]

_disconnectParser [_newclient.py:1513]

_finishResponse_WAITING [_newclient.py:1487]

dispatcher [_newclient.py:916]

_finished [_newclient.py:440]

dataReceived [http.py:1478]

rawDataReceived [_newclient.py:299]

dataReceived [basic.py:578]

dataReceived [_newclient.py:385]

dataReceived [_newclient.py:1533]

dataReceived [endpoints.py:102]

dataReceived [policies.py:120]

_flushReceiveBIO [tls.py:392]

dataReceived [tls.py:422]

_dataReceived [tcp.py:215]

doRead [tcp.py:209]

_doReadOrWrite [posixbase.py:597]

callWithContext [context.py:81]

callWithContext [context.py:118]

callWithContext [log.py:84]

callWithLogger [log.py:101]

doPoll [epollreactor.py:396]

mainLoop [base.py:1206]

run [base.py:1194]

start [crawler.py:251]

[start.py:15]

run [pydevd.py:931]

[pydevd.py:1524]

I thought that this method should be called only once for one page, why it gets called 3 times?

this brought me a problem, seems the DB access gets a pb, the result are not reflected, in the end I have 3 record in DB.

I will look into the pb, but I would still need to understand why this method gets 3 calls, did I make something wrong?

thanks and best regards