Package: twscrape
twscrapeTwitter GraphQL API implementation withSNScrapedata models.InstallpipinstalltwscrapeOr development version:pipinstallgit+https://github.com/vladkens/twscrape.gitFeaturesSupport both Search & GraphQL Twitter APIAsync/Await functions (can run multiple scrapers in parallel at the same time)Login flow (with receiving verification code from email)Saving/restoring account sessionsRaw Twitter API responses & SNScrape modelsAutomatic account switching to smooth Twitter API rate limitsUsageSince this project works through an authorized API, accounts need to be added. You can register and add an account yourself. You can also google sites that provide these things.The email password is needed to get the code to log in to the account automatically (via imap protocol).Data models:UserTweetimportasynciofromtwscrapeimportAPI,gatherfromtwscrape.loggerimportset_log_levelasyncdefmain():api=API()# or API("path-to.db") - default is `accounts.db`# ADD ACCOUNTS (for CLI usage see BELOW)awaitapi.pool.add_account("user1","pass1","u1@example.com","mail_pass1")awaitapi.pool.add_account("user2","pass2","u2@example.com","mail_pass2")awaitapi.pool.login_all()# or add account with COOKIES (with cookies login not required)cookies="abc=12; ct0=xyz"# or '{"abc": "12", "ct0": "xyz"}'awaitapi.pool.add_account("user3","pass3","u3@mail.com","mail_pass3",cookies=cookies)# API USAGE# search (latest tab)awaitgather(api.search("elon musk",limit=20))# list[Tweet]# change search tab (product), can be: Top, Latest (default), Mediaawaitgather(api.search("elon musk",limit=20,kv={"product":"Top"}))# tweet infotweet_id=20awaitapi.tweet_details(tweet_id)# Tweetawaitgather(api.retweeters(tweet_id,limit=20))# list[User]awaitgather(api.favoriters(tweet_id,limit=20))# list[User]# Note: this method have small pagination from X side, like 5 tweets per queryawaitgather(api.tweet_replies(tweet_id,limit=20))# list[Tweet]# get user by loginuser_login="xdevelopers"awaitapi.user_by_login(user_login)# User# user infouser_id=2244994945awaitapi.user_by_id(user_id)# Userawaitgather(api.following(user_id,limit=20))# list[User]awaitgather(api.followers(user_id,limit=20))# list[User]awaitgather(api.verified_followers(user_id,limit=20))# list[User]awaitgather(api.subscriptions(user_id,limit=20))# list[User]awaitgather(api.user_tweets(user_id,limit=20))# list[Tweet]awaitgather(api.user_tweets_and_replies(user_id,limit=20))# list[Tweet]awaitgather(api.liked_tweets(user_id,limit=20))# list[Tweet]# list infolist_id=123456789awaitgather(api.list_timeline(list_id))# NOTE 1: gather is a helper function to receive all data as list, FOR can be used as well:asyncfortweetinapi.search("elon musk"):print(tweet.id,tweet.user.username,tweet.rawContent)# tweet is `Tweet` object# NOTE 2: all methods have `raw` version (returns `httpx.Response` object):asyncforrepinapi.search_raw("elon musk"):print(rep.status_code,rep.json())# rep is `httpx.Response` object# change log level, default infoset_log_level("DEBUG")# Tweet & User model can be converted to regular dict or json, e.g.:doc=awaitapi.user_by_id(user_id)# Userdoc.dict()# -> python dictdoc.json()# -> json stringif__name__=="__main__":asyncio.run(main())Stoping iteration with breakIn order to correctly release an account in case ofbreakin loop, a special syntax must be used. Otherwise, Python's events loop will release lock on the account sometime in the future. See explanationhere.fromcontextlibimportaclosingasyncwithaclosing(api.search("elon musk"))asgen:asyncfortweetingen:iftweet.id<200:breakCLIGet help on CLI commands# show all commandstwscrape# help on specific comandtwscrapesearch--helpAdd accountsTo add accounts useadd_accountscommand. Command syntax is:twscrapeadd_accounts<file_path><line_format>Where:<line_format>is format of line if accounts file splited by delimeter. Possible tokens:username– requiredpassword– requiredemail– requiredemail_password– to receive email code (you can use--manualmode to get code)cookies– can be any parsable format (string, json, base64 string, etc)_– skip column from parseTokens should be splited by delimeter, usually ":" used.Example:I have account files namedorder-12345.txtwith format:username:password:email:email password:user_agent:cookiesCommand to add accounts will be (user_agent column skiped with_):twscrapeadd_accounts./order-12345.txtusername:password:email:email_password:_:cookiesLogin accountsNote:If you added accounts with cookies, login not required.Run:twscrapelogin_accountstwscrapewill start login flow for each new account. If X will ask to verify email and you providedemail_passwordinadd_account, thentwscrapewill try to receive verification code by IMAP protocol. After success login account cookies will be saved to db file for future use.Note:You can increase timeout for verification code withTWS_WAIT_EMAIL_CODEenvironment variable (default:40, in seconds).Manual email verificationIn case your email provider not support IMAP protocol (ProtonMail, Tutanota, etc) or IMAP is disabled in settings, you can enter email verification code manually. To do this run login command with--manualflag.Example:twscrapelogin_accounts--manual
twscrapereloginuser1user2--manual
twscraperelogin_failed--manualGet list of accounts and their statusestwscrapeaccounts# Output:# username  logged_in  active  last_used            total_req  error_msg# user1     True       True    2023-05-20 03:20:40  100        None# user2     True       True    2023-05-20 03:25:45  120        None# user3     False      False   None                 120        Login errorRe-login accountsIt is possible to re-login specific accounts:twscrapereloginuser1user2Or retry login for all failed logins:twscraperelogin_failedUse different accounts fileUseful if using a different set of accounts for different actionstwscrape --db test-accounts.db <command>Search commandstwscrapesearch"QUERY"--limit=20twscrapetweet_detailsTWEET_ID
twscrapetweet_repliesTWEET_ID--limit=20twscraperetweetersTWEET_ID--limit=20twscrapefavoritersTWEET_ID--limit=20twscrapeuser_by_idUSER_ID
twscrapeuser_by_loginUSERNAME
twscrapefollowingUSER_ID--limit=20twscrapefollowersUSER_ID--limit=20twscrapeverified_followersUSER_ID--limit=20twscrapesubscriptionsUSER_ID--limit=20twscrapeuser_tweetsUSER_ID--limit=20twscrapeuser_tweets_and_repliesUSER_ID--limit=20twscrapeliked_tweetsUSER_ID--limit=20The default output is in the console (stdout), one document per line. So it can be redirected to the file.twscrapesearch"elon mask lang:es"--limit=20>data.txtBy default, parsed data is returned. The original tweet responses can be retrieved with--rawflag.twscrapesearch"elon mask lang:es"--limit=20--rawProxyThere are few options to use proxies.You can add proxy per accountproxy="http://login:pass@example.com:8080"awaitapi.pool.add_account("user4","pass4","u4@mail.com","mail_pass4",proxy=proxy)You can use global proxy for all accountsproxy="http://login:pass@example.com:8080"api=API(proxy=proxy)doc=awaitapi.user_by_login("elonmusk")Use can set proxy with environemt variableTWS_RPOXY:TWS_PROXY=socks5://user:pass@127.0.0.1:1080twscrapeuser_by_loginelonmuskYou can change proxy any time like:api.proxy="socks5://user:pass@127.0.0.1:1080"doc=awaitapi.user_by_login("elonmusk")# new proxy will be usedapi.proxy=Nonedoc=awaitapi.user_by_login("elonmusk")# no proxy usedProxy prioritiesapi.proxyhave top priorityenv.proxywill be used ifapi.proxyis Noneacc.proxyhave lowest priotitySo if you want to use proxy PER ACCOUNT, do NOT override proxy with env variable or by passing proxy param to API.Note:If proxy not working, exception will be raised from API class.LimitationsAfter 1 July 2023 Twitterintroduced new limitsand still continue to update it periodically.The basic behaviour is as follows:the reqest limit is updated every 15 minutes for each endpoint individuallye.g. each account have 50 search requests / 15 min, 50 profile requests / 15 min, etc.API data limits:user_tweets&user_tweets_and_replies– can return ~3200 tweets maximumArticlesHow to still scrape millions of tweets in 2023(Add Article)See alsotwitter-advanced-search– guide on search filterstwitter-api-client– Implementation of Twitter's v1, v2, and GraphQL APIssnscrape– is a scraper for social networking services (SNS)twint– Twitter Intelligence Tool