Package: ntscraper
Unofficial Nitter scraperNoteTwitter has recently made some changes which affected every third party Twitter client, including Nitter. As a result, most Nitter instances have shut down or will shut down shortly. Even local instances are affected by this, so you may not be able to scrape as many tweets as expected, if at all.The scraperThis is a simple library to scrape Nitter instances for tweets. It can:search and scrape tweets with a certain termsearch and scrape tweets with a certain hashtagscrape tweets from a user profileget profile information of a user, such as display name, username, number of tweets, profile picture ...If the instance to use is not provided to the scraper, it will use a random public instance. If you can, please host your own instance in order to avoid overloading the public ones and letting Nitter stay alive for everyone. You can read more about that here:https://github.com/zedeus/nitter#installation.Installationpip install ntscraperHow to useFirst, initialize the library:fromntscraperimportNitterscraper=Nitter(log_level=1,skip_instance_check=False)The valid logging levels are:None = no logs0 = only warning and error logs1 = previous + informational logs (default)Theskip_instance_checkparameter is used to skip the check of the Nitter instances altogether during the execution of the script. If you use your own instance or trust the instance you are relying on, then you can skip set it to 'True', otherwise it's better to leave it to false.Then, choose the proper function for what you want to do from the following.Scrape tweetsgithub_hash_tweets=scraper.get_tweets("github",mode='hashtag')bezos_tweets=scraper.get_tweets("JeffBezos",mode='user')Parameters:term: search termmode: modality to scrape the tweets. Default is 'term' which will look for tweets containing the search term. Other modes are 'hashtag' to search for a hashtag and 'user' to scrape tweets from a user profilenumber: number of tweets to scrape. Default is -1 (no limit).since: date to start scraping from, formatted as YYYY-MM-DD. Default is Noneuntil: date to stop scraping at, formatted as YYYY-MM-DD. Default is Nonenear: location to search tweets from. Default is None (anywhere)language: language of the tweets to search. Default is None (any language). The language must be specified as a 2-letter ISO 639-1 code (e.g. 'en' for English, 'es' for Spanish, 'fr' for French ...)to: user to which the tweets are directed. Default is None (any user). For example, if you want to search for tweets directed to @github, you would set this parameter to 'github'filters: list of filters to apply to the search. Default is None. Valid filters are: 'nativeretweets', 'media', 'videos', 'news', 'verified', 'native_video', 'replies', 'links', 'images', 'safe', 'quote', 'pro_video'exclude: list of filters to exclude from the search. Default is None. Valid filters are the same as abovemax_retries: max retries to scrape a page. Default is 5instance: Nitter instance to use. Default is None and will be chosen at randomReturns a dictionary with tweets and threads for the term.MultiprocessingYou can also scrape multiple terms at once using multiprocessing:terms=["github","bezos","musk"]results=scraper.get_tweets(terms,mode='term')Each term will be scraped in a different process. The result will be a list of dictionaries, one for each term.The multiprocessing code needs to run in aif __name__ == "__main__"block to avoid errors. With multiprocessing, only full logging is supported. Also, the number of processes is limited to the number of available cores on your machine.NOTE: using multiprocessing on public instances is highly discouraged since it puts too much load on the servers and could potentially also get you rate limited. Please only use it on your local instance.Get profile informationbezos_information=scraper.get_profile_info("JeffBezos")Parameters:username: username of the page to scrapemax_retries: max retries to scrape a page. Default is 5instance: Nitter instance to use. Default is NoneReturns a dictionary of the profile's information.MultiprocessingAs for the term scraping, you can also get info from multiple profiles at once using multiprocessing:usernames=["x","github"]results=scraper.get_profile_info(usernames)Each user will be scraped in a different process. The result will be a list of dictionaries, one for each user.The multiprocessing code needs to run in aif __name__ == "__main__"block to avoid errors. With multiprocessing, only full logging is supported. Also, the number of processes is limited to the number of available cores on your machine.NOTE: using multiprocessing on public instances is highly discouraged since it puts too much load on the servers and could potentially also get you rate limited. Please only use it on your local instance.Get random Nitter instancerandom_instance=scraper.get_random_instance()Returns a random Nitter instance.NoteDue to recent changes on Twitter's side, some Nitter instances may not work properly even if they are marked as "working" on Nitter's wiki. If you have trouble scraping with a certain instance, try changing it and check if the problem persists.To do listAdd scraping of individual posts with comments