Package: scrapy-sessions
scrapy-sessionsA session-management extension for Scrapy.OverviewThis library resolves at least three long-standing issues in Scrapy's session-management system that people have raised concerns about for years:Scrapy's sessions are effectively a black box. They are difficult to expose and alter within a scrape.Scrapy makes it very difficult to replace/refresh a session (and/or general 'profile') unilaterally across all requests that are scheduled or enqueued. This is important for engaging with websites that have a session-expiry system based on profile (IP/user-agent) or use short-lived sessions that require a custom renewal logic. Scrapy's cookie system fails to handle such websites.Scrapy provides no native capability for maintaining distinct profiles (client identities) within a single scrape.This library contains aCookiesMiddlewarethat exposes the Scrapy cookie jars in the spider attributesessions. This is an instance of the newSessionsclass (objects.Sessions) that allows one to examine the content of the current sessions and to clear and/or renew a session that is failing. The renewal procedure short-circuits the Scrapy request scheduling process, inducing an immediate download of the request specified, ahead of all others. This does not cause any adverse consequences (for example, scrape statistics are maintained perfectly).This library also provides a tool for maintaining and rotating "profiles", making it easy to give the appearance that your scrape's requests are being generated by multiple, entirely distinct clients.Another use case is for handling session cookies collected outside of Scrapy and fed into your spider. Whenever this external collection is necessary (for websites that require some kind of demonstration of Javascript rendering before they serve a session to an unknown client), this library provides a handy solution for cycling from one session to the next at each point of failure.Relation to the Default Scrapy CookiesMiddlewareThescrapy-sessionsCookiesMiddlewareis designed to override the default ScrapyCookiesMiddleware. It is anextensionof the default middleware, so there shouldn't be adverse consequences from adopting it.The"COOKIES_ENABLED"and"COOKIES_DEBUG"settings work exactly as with the default middleware: if"COOKIES_ENABLED"is disabled, this middleware is disabled, and if"COOKIES_DEBUG"is enabled, you will get the same debug messages about cookies sent and received.With this said, there are some important differences to note. With the default Scrapy middleware, the value of the"cookiejar"key in yourrequest.metanames the session (cookie jar) that the request will use. If the session does not exist, a new session is created. The exact same applies in this library, except that you can now also use the"session_id"key for this purpose. The default value for this is now0, rather thanNone. So, if you don't use either of these keywords in any of your requests, each request will by default send the cookies associated with session0, and add any cookies it receives to session0.Set upBasicOverride the default middleware:DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
    'scrapy_sessions.CookiesMiddleware': 700,
}This will allow you to interact with thespider.sessionsattribute, in order to inspect, modify, clear, and renew sessions (seeusage). It will also give you access to the response cookies viaresponse.meta["cookies"].ProfilesThis is a separate add-on that hooks onto the sessions.After changingsettings.pyas above, add the following:SESSIONS_PROFILES_SYNC: True.Then create aprofiles.pyfile at the head of your project similar to the following:from w3lib.http import basic_auth_header
PROFILES = [
    {"proxy":['proxy_url', basic_auth_header('username', 'password')], "user-agent": "MY USER AGENT"},
    {"proxy":['proxy_url', basic_auth_header('username', 'password')], "user-agent": "MY USER AGENT"}
](Either the "proxy" key or the "user-agent" key can be omitted for each profile (but not both).)Finally, after importing theload_profilesfunction (from scrapy_sessions.utils import load_profiles), add the following to your spider settings:custom_settings = {
  "SESSIONS_PROFILES":load_profiles('profiles.py')
}Currently, thisload_profilesfunction fails when trying to deploy on Zyte. I will try to solve this issue when I have time.MethodsAccess these methods via thesessionsinstance attached to your spider when using this library. For example usage, see the next section.getSignatureget(self, session_id=0, mode=None, domain=None)DescriptionFor inspecting your sessions. Two formats toggled by mode: dictionary (name:value) or list of strings (containing the full cookie data for each cookie).get_profileSignatureget_profile(self, session_id=0)DescriptionFor inspecting the profile attached to the given session. Only works ifSESSIONS_PROFILES_SYNCis enabled.add_cookies_manuallySignatureadd_cookies_manually(self, cookies, url, session_id=0)DescriptionFor explicitly adding a set of cookies to a given session. The cookies must be in the format {name:value}; the url is the url that these cookies would come from conventionally.clearSignatureclear(self, session_id=0, renewal_request=None)DescriptionFor clearing a session and/or immediately renewing it with a special one-off request. If you don't specify a renewal_request, the session will be retried with the first new request off the rank.UsageAccessing the cookies received in the last responseresponse.meta["cookies"]Accessing the current session idresponse.meta["session_id"]In the belowselfis referring to aScrapy.spiderclass.Viewing a sessionThe cookies in the first domain of the default session (session 0):self.sessions.get()The cookies in the first domain of a specified session:self.sessions.get(response.meta["session_id"])A specified session with a specified domain:self.sessions.get(response.meta["session_id"], domain='exampledomain.com')In dictionary format:self.sessions.get(session_id, mode=dict)Clearing a sessionThe default session:self.sessions.clear()Specifying a session works the same as inget.Clearing and immediately renewing a session (instantly downloaded out of sync)The default session:self.sessions.clear(renewal_request=Request(url='renewal_url',callback=self.cb))The callback is optional; if no callback is specified, the session is renewed just the same.Viewing a profileThe profile for the default session:self.sessions.get_profile()Specifying a session works the same as before.This method will only work ifSESSIONS_PROFILES_SYNCis enabled in the spider settings.Explicitly adding cookies to a sessionself.sessions.add_cookies_manually({name1: val1, name2: val2}, 'https://exampledomain.com/', 0)Session Refresh MotivationThere are two use cases for this:For handling websites that track session usage by some aspect of client identity, such as IP. This is not a common web-security feature but it does exist, and Scrapy can't handle it. By default, Scrapy will send all your requests with the one session, so if you send all your requests with the same identity signatures also, then you will be able to navigate such sites until your session expires due to reaching a time or usage limit. When this session expires, though, you need to refresh it and initiate a new one with a new identity. This library provides two ways of doing this, with and without using theProfilesadd-on.For handling the rotation of sessions collected by some process external to Scrapy. You might make use of such an approach whenever you are unable to collect a valid session without being able to render Javascript on a site, as in the case of sites that validate clients based on fingerprinting techniques. However you collect these sessions, it is vital to be able to seamlessly switch from session A to session B as soon as session A starts failing; the clear-and-renew amenity provided by this package is the appropriate solution.Session Refresh ImplementationWith ProfilesSet up your profiles, then within some part of an errback function or middleware that only gets activated when a session expires (you may need some custom logic here), clear and renew your session usingsessions.clear. Because you are usingprofiles, then anyrenewal_requestyou specify within theclearmethod will automatically get visited by a fresh profile.Without ProfilesWithin some part of an errback function or middleware that only gets activated when a session expires, clear and renew your session usingsessions.clearby specifying arenewal_requestthat uses a fresh proxy and/or user-agent.Session Refresh LogicSince this is the most complicated part of the library it's worth describing the underlying process. The following is what happens whenclearis called with arenewal_requestargument:The specified session is cleared and the request specified is immediately downloaded, without entering the standard request queue. The way I have achieved this, the logs and statistics are updated as normal and everything seems to go smoothly.The response derived from therenewal_requestwill reach theprocess_responsemethod of the middleware and therein re-fill the session.Any other requests/responses that reach the middleware after thecleartrigger but before the renewal event has occurred are re-scheduled or re-downloaded. To see the manner in which this is achieved, see the code.ProfilesThe idea of this tool is to manage distinct client identities within a scrape. The identity consists of two or more of the following attributes: session + user agent + proxy.The profiles are input via a specialprofiles.pyfile (seesetting up profiles). Once you have these set up (and have tweaked the settings as required), one of these profiles is automatically associated with every new session created in your scrape. If there are more sessions than profiles, the profiles will be automatically recycled from the beginning. When a session is cleared, the profile is also removed.How it worksIndex 0 of any "proxy" value is fed into therequest.meta["proxy"]field in theprocess_requestfunction of the middleware. Index 1 is fed intorequest.headers['Proxy-Authorization'].Similarly, the "user-agent" value is fed intorequest.headers["user-agent"].Future DirectionsI am planning to add tests, and then I may at some point submit a pull request on the Scrapy repository proposing this as a replacement for the default ScrapyCookiesMiddleware.MiscellaneousI've noticed what might be described as a bug in the default Scrapy implementation of the cookiejar via the http.cookiejar library. I'm not sure it raises to the level of a bug but either way it's an unexpected behaviour. This library has not addressed it because it would be best addressed within the http.cookiejar library itself. The behaviour is as follows:In Scrapy, you can send off a number of cookies with a single request. These get merged with the existing cookies in the session before the request is sent off. One way of adding these cookies to the request is in list format, and within this format, you can specify a domain for each cookie. (e.g.cookies=[{'name':name','value':value,'domain':domain}]). If you specify this domain, these cookies will get stored in the session (cookiejar) under that domain except with a leading dot (full stop). It doesn't matter whether you include this leading dot yourself or not. But, by default, any cookies that get added to the session when you make requests to that very same domain without explicit cookies will be added to the session without this leading dot. Therefore, because these two sets of cookies only get merged together if they are filed under an identical domain, the cookies may fail to merge properly in the next request to that domain, so that you end up sending cookies that overlap (with duplicate names or potentially even duplicate names + values).This can be resolved either by simply not adding an explicit domain to cookies that you specify in this way, or by using the methodadd_cookies_manuallyto add these extra cookies to the session before you send off any requests that require them.