Package: autologin-middleware
This is a a Scrapy middleware that usesautologinhttp-api
to maintain a logged-in state for a scrapy spider.Autologin middleware uses autologin to make all requests while being
logged in. It uses autologin to get cookies, detects logouts and tries
to avoid them in the future. A single authorization domain for the spider
is assumed. Autologin middleware also putsautologin_activeintorequest.meta, which isTrueonly if we are logged in
(and toFalseif domain is skipped or login failed).
If requests are made viasplash(andSPLASH_URLis set),
autologin middleware passes it to autologin,
and this splash instance is also used to obtain login cookies.InstallationIt works on python 2.7 and python 3, and requires at least scrapy 1.1.
Install with pip:pip install autologin-middlewareConfigurationInclude the autologin middleware into the project settings
and specify autologin url:AUTOLOGIN_URL = 'http://127.0.0.1:8089'
AUTOLOGIN_ENABLED = True
DOWNLOADER_MIDDLEWARES['autologin_middleware.AutologinMiddleware'] = 605Cookie support is also required. There are currently several options:scrapy cookie middleware (COOKIES_ENABLED = True),
but autologin middleware requires access to cookies, so you need to enable
a custom cookie middleware:DOWNLOADER_MIDDLEWARES = {
    'autologin_middleware.AutologinMiddleware': 605,
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
    'autologin_middleware.ExposeCookiesMiddleware': 700,
}scrapy-splashcookie middleware (scrapy_splash.SplashCookiesMiddleware)any other middleware that gets cookies fromrequest.cookiesand
setsresponse.cookiejarlike scrapy-splash middleware,
or exposes them inresponse.flagslikeExposeCookiesMiddleware.Optional but highly recommended: avoid logouts - see optional settings and
“Avoiding logouts” section below. The reason why it’s important is that
logout detection is less robust, on some sites the old cookies might still
work so you’ll be using multiple sessions, etc. If you don’t ever logout,
you avoid all this problems.There are some optional settings:AUTOLOGIN_COOKIES: pass auth cookies after manual login
(format isname=value; name2=value2).AUTOLOGIN_LOGOUT_URL: pass url substring to avoid.AUTOLOGIN_CHECK_LOGOUT: set toFalsein order to disable automatic
logout detection: it remembers cookies obtained during login and
checks them on each response to see if any disappeared. This can be
problematic for sites that set a lot of cookies on login,
so this is an option to disable it.
If you disable it, you must rely on avoiding logout links withlink_looks_like_logout(see below), or setting a customAUTOLOGIN_LOGOUT_URL.AUTOLOGIN_USERNAME,AUTOLOGIN_PASSWORD,AUTOLOGIN_LOGIN_URL,AUTOLOGIN_EXTRA_JSare passed to autologin and override values
from stored credentials.AUTOLOGIN_LOGIN_URLis a relative url,
and can be omitted if it is the same as the start url.AUTOLOGIN_EXTRA_JSis required only if you want to use theextra_jsfeature of the autologin.It is also possible to override some settings per-request via corresponding
lower-case keys inrequest.meta:autologin_username,autologin_password,autologin_extra_js,autologin_login_urlandautologin_logout_url.Autologin middleware passes the following settings to the autologin:SPLASH_URL,USER_AGENT,HTTP_PROXY,HTTPS_PROXY, so they
are used for autologin requests.Avoiding logoutsThere is also an utilityautologin_middleware.link_looks_like_logoutfor checking if a links looks like a logout link: you can use it in the
spider to avoid logout links. Logouts are handled
by the autologin middleware by default
(unlessAUTOLOGIN_CHECK_LOGOUTisFalse),
but avoiding logout links can be beneficial for two reasons:no time is waster retrying requests that were logged outin some cases, logout urls can be unique, and the spider will be logging
out continuously (for example,/logout?sid=UNIQUE_ID).APIThere is no special API: autologin middleware just ensures that all requests are
made while being logged in. As mentioned in the “Configuration” section above,
you can override some settings on the per-request basis inreqeuest.meta.Autologin response is available inresponse.meta['autologin_response'],
if we made requests to autologin while processing this request.
You might want to use the"status"field of the autologin response
to do some bookkeeping.Middleware also always puts"autologin_active"intoresponse.meta,
which isTrueonly if we are logged in (andFalseif domain is skipped
or login failed).Additionally, you setskip_autologinkey inrequest.metatoTruein order to completely disable middleware for this request.Usage with SplashAutologin middleware supports splash viascrapy-splash,
but correctly settings everything up can be tricky.First, you need to specify the following settings
(check scrapy-splash docs for more details):SPLASH_URL = 'http://127.0.0.1:8050'
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
DOWNLOADER_MIDDLEWARES = {
    'autologin_middleware.AutologinMiddleware': 605,
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}Second, you need to make requests to splash and pass cookies withsplash:init_cookies(splash.args.cookies), and return them in thecookiesfield usingsplash:get_cookies(). If you are already using
a splash script (executeendpoint), modify your script accordingly.
But if you just want to crawl using splash, you can useautologin_middleware.splash.splash_requestinstead ofscrapy.Request.
It has a minimal lua script that passes cookies and returns html, so you won’t
need to change anything else in you spider.DevelopmentYou need to startautologin-http-api(fromautologin),
andsplash(the easiest option is to rundocker run-p8050:8050 scrapinghub/splash).Run tests with tox:toxWhen using Docker to run Splash on OS X and Windows, it will start on
a non-default address, so you need to specify it when running tests,
for example:SPLASH_URL=http://192.168.99.100:8050 toxLicenseLicense is MIT.