Package: twiget
TwiGetTwiGet is a python package for the management of the queries on filtered stream of the Twitter API, and the collection of tweets from it.It can be used as a command line tool (twiget-cli) or as a python class (TwiGet).Installation> pip install twigetThe command installs the package and also makes thetwiget-clicommand available.Command line tool: twiget-cliTwiGet implements a command line interface that can be started with the command:> twiget-cliWhen launched without arguments the program searches for a.twiget.conffile in theHOMEdirectory (the directory pointed by the$HOMEor%userprofile%environment variable).
The file must contain in the first line thebearer tokenthat allows the program to access the Twitter API.Alternatively, the name of the file from which to obtain the bearer token can be given as argument when starting the program:> twiget-cli -b path_to_file/with_token.txtNOTE: store the bearer token in a file with minimum access permissions. Never share it. Revoke any tokens that may have been made public.Another optional argument is the path where to save collected tweets.
By default, adatasubdirectory is created in the current working directory.> twiget-cli -s ./save_dirpromptWhen started, twiget-cli shown the available commands, and the queries currently registered for the given bearer token (queries are permanently stored on Twitter's servers).TwiGet 0.1.1

Available commands (type help <command> for details):
create, delete, exit, help, list, refresh, save_to, size, start, stop

Registered queries:
	ID=1385892384573355842	query="#usa"	tag="usa"
	ID=1405490304970434817	query="bts"	tag="bts"The command prompt tells if twiget-cli is currently collecting tweets, the number of collected tweets, and the save path.[not collecting (0 since last start), save path "data"]>When collecting tweets, the prompt is automatically refreshed every time a given number of tweets is collected (seethe refresh command).CommandscreateFormat:> create <tag> <query>Creates a filtering rule, associated to a given tag name.Collected tweets are saved in json format in a file named<tag>.json, in the given save path.
Tag name is the first argument of the command and cannot contain spaces.Any word after the tag defines the query.Info on how to define rules.Example:[not collecting (0 since last start), save path "data"]>create usa jow biden
Tweets matching the query "jow biden" will be saved in the file data/usa.json
ID=1395720345987340524listFormat:> listLists the queries, their ID and their tag, currently registered for the filtered stream.Example:[not collecting (0 since last start), save path "data"]> list
Registered queries:
	ID=1385892384573355842	query="#usa"	tag="usa"
	ID=13905490304970434817	query="bts"	tag="bts"
	ID=1395720345987340524	query="joe biden"	tag="usa"deleteFormat:> delete <ID>Deletes a query, given its ID.Example:[not collecting (0 since last start), save path "data"]> delete 1385892384573355842startFormat:> startStarts a background process that collects tweets from the filtered stream and puts them in json files, according to the tag they are associated to.Collection continues until astopor aexitcommand is entered.
To let TwiGet collect data for longer periods of time, I suggest to use TwiGet within a virtual terminal session, using, e.g.,screenortmux.Note: create and delete command can be issued also when collecting tweets. The collection process is updated immediately.Example:[not collecting (0 since last start), save path "data"]> start
[collecting (0 since last start), save path "data"]>stopFormat:> stopStop data collections.Example:[collecting (3000 since last start), save path "data"]> stop
[not collecting (3152 since last start), save path "data"]>save_toFormat:> save_to <path>Sets the path where json files are saved._Note: changing path while collecting tweets will immediately create new json file in the new path, leaving all tweets collected until that moment in the old path.Example:[not collecting (0 since last start), save path "data"]> save_to ../my_project
[not collecting (0 since last start), save path "../my_project"]>sizeFormat:> size <size>Sets the maximum size in bytes of json files.
When a json file reaches this size, a new file with an incremented index (e.g., tag_0.json, tag_1.json, tag_2.json...) is created.Example:[not collecting (0 since last start), save path "data"]> size 1000000refreshFormat:> refresh <count>Sets the number of collected tweets that triggers an automatic refresh of the prompt.Example:[not collecting (0 since last start), save path "data"]> refresh 10000Implementing a custom command line toolTheTwiGetCLIBaseclass intwiget_cli.pymodule implements all the above fuctions except those related to saving to json file (i.e.save_toandsize).
It can be used to implement a command line tool that performs a different processing of the collected data, e.g., saving to a db.Python class: TwiGetTwiGet core functionalities are implemented in a python class, which can be directly used in python code.fromtwigetimportTwiGetbearer='put here the bearer token'collector=TwiGet(bearer)# Adding a filtering rule# https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/post-tweets-search-stream-rulesquery='support vector machine'tag='ml'answer=collector.add_rule(query,tag)# returns the parsed json answer from the server.print(answer)# Listing the current filtering rules# https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream-rulesanswer=collector.get_rules()# returns the parsed json answer from the server.print(answer)# Delete some rules by giving their ID# https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/post-tweets-search-stream-rulesids=[48573094587309485,3029834285720978]answer=collector.delete_rules(ids)# returns the parsed json answer from the server.print(answer)# Adding a callback# The data argument contains the content and information about the retrieved tweet# https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-streamdefprint_tag(data):print(data['matching_rules'][0]['tag'])answer=collector.add_callback('print tag',print_tag)# returns the parsed json answer from the server.print(answer)# Getting callbackscallbacks=collector.get_callbacks()# returns a list of pairs with the name of the callback and the callback method.print(callbacks)# Delete a callbackcollector.delete_callback('print tag')# Starting tweet collectioncollector.start_getting_stream()# Checking status of collectionrunning=collector.is_getting_stream()# returns a boolean. True if collection is active.print(running)# Stopping tweet collectioncollector.stop_getting_stream()LicenseAuthor:Andrea EsuliBSD 3-Clause License, seelicense file