Package: autologin
Autologin is a library that makes it easier for web spiders tocrawl websites that require login.
Provide it with credentials and a URL or the html source of a page
(normally the homepage), and it will attempt to login for you.
Cookies are returned to be used by your spider.The goal of Autologin is to make it easier for web spiders to crawl websites
that require authenticationwithout having to re-write login code for each website.Autologin can be used as a library, on the command line, or as a service.
You can make use of Autologin without generating http requests,
so you can drop it right into your spider without worrying about
impacting rate limits.If you are usingScrapyfor crawling, check outautologin-middleware,
which is a scrapy middleware that uses autologin http-api to maintain
a logged-in state for a scrapy spider.Autologin works on Python 2.7 and 3.3+.NoteThe library is in the alpha stage.
API can still change, especially around the keychain UI.ContentsFeaturesQuickstartInstallationAuth cookies from URLLogin requestCommand LineHTTP APIProxy supportCaptcha supportKeychain UIContributorsLicenseFeaturesAutomatically find login forms and fieldsObtain authenticated cookiesObtain form requests to submit from your own spiderExtract links to login pagesUse as a library with or without making http requestsCommand line clientWeb serviceUI for managing login credentialsCaptcha supportQuickstartDon’t like reading documentation?from autologin import AutoLogin

url = 'https://reddit.com'
username = 'foo'
password = 'bar'
al = AutoLogin()
cookies = al.auth_cookies_from_url(url, username, password)You now have acookiejarthat you can use in your spider.  Don’t want a cookiejar?cookies.__dict__You now have a dictionary.InstallationInstall the latest release from PyPI:$ pip install -U autologinor the version with the latest changes from Github:$ pip install git+https://github.com/TeamHG-Memex/autologin.gitAutologin depends onFormasaurusfor field and form classification, which has quite a lot of dependencies.
These packages may require extra steps to install, so the command above
may fail.
In this case install dependencies manually, one by one
(follow their install instructions).A recentpipis recommended (update it withpip install pip-U).
On Ubuntu, the following packages are required:$ apt-get install build-essential libssl-dev libffi-dev \
                  libxml2-dev libxslt1-dev \
                  python-dev  # or python3-dev for python 3Auth cookies from URLThis method makes an http request to the URL,
extracts the login form (if there is one),
fills the fields and submits the form.
It then return any cookies it has picked up:cookies = al.auth_cookies_from_url(url, username, password)Note that it returns all cookies, they may be session cookies rather
than authenticated cookies.This call is blocking, and uses Crochet to run the Twisted reactor
and a Scrapy spider in a separate thread.
If you have a Scrapy spider (or use Twisted in some other way),
use the HTTP API, or the non-blocking API (it’s not documented,
seehttp_api.AutologinAPI._login).There are also optional arguments forAutoLogin.auth_cookies_from_url:settingsis a dictionary with Scrapy settings to override.
Useful settings to pass include:HTTP_PROXY,HTTPS_PROXYset proxies to use for all requests.SPLASH_URLif set,Splashwill be used to make all requests. Use it if your crawler also uses
splash and the session is tied to IP and User-Agent, or for Tor sites.USER_AGENToverrides default User-Agent.extra_js(experimental)
is a string with an extra JS script that should be executed
on the login page before making a POST request. For example, it can be used
to accept cookie use. It is supported only whenSPLASH_URLis also given
insettings.An example of using this options:cookies = al.auth_cookies_from_url(
    url, username, password,
    extra_js='document.getElementById("accept-cookies").click();',
    settings={
        'SPLASH_URL': 'http://127.0.0.1:8050',
        'USER_AGENT': 'Mozilla/2.02 [fr] (WinNT; I)',
    })Login requestThis method extracts the login form (if there is one),
fills the fields and returns a dictionary with the form url and args
for your spider to submit. No http requests are made:>>> al.login_request(html_source, username, password, base_url=None)
{'body': 'login=admin&password=secret',
 'headers': {b'Content-Type': b'application/x-www-form-urlencoded'},
 'method': 'POST',
 'url': '/login'}Relative form action will be resolved against thebase_url.Command Line$ autologin
usage: autologin [-h] [--splash-url SPLASH_URL] [--http-proxy HTTP_PROXY]
                 [--https-proxy HTTPS_PROXY] [--extra-js EXTRA_JS]
                 [--show-in-browser]
                 username password urlHTTP APIYou can start the autologin HTTP API with:$ autologin-http-apiand use/login-cookiesendpoint. Make a POST request with JSON body.
The following arguments are supported:url(required): url of the site where we would like to loginusername(optional): if not provided, it will be fetched from the
login keychainpassword(optional): if not provided, it will be fetched from the
login keychainextra_js(optional, experimental)
is a string with an extra JS script that should be executed
on the login page before making a POST request. For example, it can be used
to accept cookie use. It is supported only whenSPLASH_URLis also given
insettings.settings(optional) - a dictionary with Scrapy settings to override,
useful values are described above.Ifusernameandpasswordare not provided, autologin tries to find
them in the login keychain. If no matching credentials are found (they are
matched by domain, not by precise url), then human is expected to eventually
provide them in the keychain UI, or mark domain as “skipped”.Response is JSON with the following fields:status, which can take the following values:errorstatus means an error occurred,errorfield has more infoskippedmeans that domain is marked as “skipped” in keychain UIpendingmeans there is an item in keychain UI (or it was just created),
and no credentials have been entered yetsolvedmeans that login was successful and cookies were obtainederror- human-readable explanation of the error.response- last response received by autologin (can be None in some cases).
This is a dict withcookies,headers, and either atextorbody_b64fields (depending on response content type).cookies- a list of dictionaries inCookie.__dict__format. Present only
if status issolved.start_url- a url that was reached after successful login.Proxy supportProxies can be specified viaHTTP_PROXYandHTTPS_PROOXYkeys
insettingsargument. Username and password can be specified
as part of the proxy url (the format isprotocol://username:password@url).If you are using proxy with Splash, it is assumed that
you want to have Splash make requests via given proxy,
and not make a requesttoSplash via proxy.HTTP_PROXYis always used for Splash.Captcha supportThere is experimental captcha support: if the login form contains a captcha,
we will try to solve it using an external service (DeathByCaptcha),
and will submit it as part of login request. This does not affect API in any
way, you only have to provide environment variables with your DeathByCaptcha
account details:DEATHBYCAPTCHA_USERNAMEandDEATHBYCAPTCHA_PASSWORD.
This applies to all APIs:autologin-http-api,autologin, and
the Python API.You also need to install thedecaptchalibrary:pip install git+https://github.com/TeamHG-Memex/decaptcha.gitSupport is still experimental, new Google ReCaptcha/NoCaptcha are not supported.
Also, it currently works only with splash (whenSPLASH_URLis passed insettings).Keychain UIStart keychain UI with:$ autologin-serverNote that bothautologin-serverandautologin-http-apiare not protected by any authentication.Keychain UI stores credentials in an sqlite database. It is located near
the library itself by default, which is not always good, especially if you want
to persist the data between updates or do not have write permissions
for that folder. You can configure database location andsecret_keyused by the flask app by creating an/etc/autologin.cfgor~/.autologin.cfgfile (should be the same user under which autologin
services are running). Here is an example config that changes default secret_key
and specifies a different database path (both items are optional):[autologin]
secret_key = 8a0b923820dcc509a6f75849b
db = /var/autologin/db.sqliteContributorsSource code and bug tracker are on github:https://github.com/TeamHG-Memex/autologin.Run tests withtox:$ toxSplash support is not tested directly here, but there are indirect tests for it
in theautologin-middlewaretest suite.LicenseLicense is MIT.