Package: tidy-tweet
tidy_tweetTidies Twitter json collected with Twarc into relational tables.The resulting SQLite database is ideal for importing into analytical tools, or for using as a datasource for a
programmatic analytical workflow that is more efficient than working directly from the raw JSON. However, we always
recommend retaining the raw JSON data - think of tidy_tweet and its resulting databases as the first step of data
pre-processing, rather than as the original/raw data for your project.WARNING- tidy_tweet is still released in a preliminary version, not all data fields are loaded into the database,
and we can't guarantee no breaking changes either of library interface or database schema before 1.0 release. Most
notably, the database schema will have a significant change to allow multiple JSON files to be loaded into the same
database file.ContentsCollecting Twitter DataInput and OutputPrerequisitesInstallationUsageFeedback and ContributionsAbout tidy_tweetCollecting Twitter dataIf you do not have a preferred Twitter collection tool already, we recommendTwarc.
tidy_tweet is designed to work directly with Twarc output. Other collection methods may work with tidy_tweet as long
as they output the API result from Twitter with minimal alteration (seeInput and Output), however
at this time we do not have the resources to support Twitter data outputs from tools other than Twarc.Input and OutputInput: Twitter tweet results pagestidy_tweet takes as input a series of JSON/dict objects, each object of which is a page of Twitter API v2 search or
timeline results. Typically, this will be a JSON file such as those output bytwarc2 search. At present, API endpoints
oriented around things other than tweets, such as theliking-usersendpoint, are not properly supported, though we
hope to support them in future.JSON files with multiple pages of results are expected to be newline-delimited, with each line being a distinct results
page object, and no commas between top-level objects.Output: Sqlite database of tweets and metadataAfter processing your Twitter results pages with tidy_tweet (seeUsage), you will have anSQLitedatabase file at the location you specified.Database schema will be published here as soon as the initial schema is finalised.PrerequisitesPython 3.8+A command line shell/terminal, such as bash, Mac Terminal, Git Bash, Anaconda Prompt, etcThis tool requires Python 3.8 or later, the instructions assume you already have Python installed. If you haven't
installed Python before, you might findPython for Beginnershelpful - note that tidy_tweet is a
command line application, you don't need to write any Python code to use it (although you can if you want to), you just
need to be able to run Python code!The instructions assume sufficient familiarity with using a command line to change directories, list files and find
their locations, and execute commands. If you are new to the command line or want a refresher, there are some good
lessons fromSoftware Carpentryand theProgramming Historian.The instructions assume you are working in a suitable Pythonvirtual environment. RealPython has a relatively straightforwardprimer on virtual environmentsif you are new to the concept. If you installed Python with
Anaconda/conda, you will want to manage your virtual environments throughAnaconda/condaas well. If you have a virtual environment already set up for usingTwarc, you can install tidy_tweet in that
same environment.Installationtidy_tweet is a Python package and can be installed with pip.Ensure you are using an appropriate Python or Anaconda environment (seePrerequisites)Install tidy_tweet and its requirements by running:python-mpipinstalltidy_tweetRun the following to check that your environment is ready to run tidy_tweet:tidy_tweet--helpIf you wish to install a specific version of tidy_tweet, for example to replicate past results, you can specify the
desired version when installing with pip, for example to install tidy_tweet version 1.0.1 (which does not currently
exist):python-mpipinstalltidy-tweet==1.0.1Usagetidy_tweet may be used either as acommand line applicationor as
aPython library. The command line interface (CLI) is recommended for general use and is intended to
be more straightforward to use. The Python library interface is designed for use cases such as integrating tidy_tweet
usage into other tools, scripts, and notebooks.Command line interfaceAfterinstalling tidy_tweet, you should be able to runtidy_tweetas a command line application:tidy_tweet--helpRunning the above will show you a summary of how to use the tidy_tweet command line interface (CLI). The
tidy_tweet CLI expects you to provide specific arguments in a specific order, as follows:tidy_tweetDATABASEJSON_FILEDATABASE: This is the filename where you want to save the tidied data as a database. As this is anSQLitedatabase, it is conventional for the filename to end in ".db". Example:my_dataset.dbJSON_FILE: This is the file of tweets you wish to tidy into the database. For more information,
seeInput and OutputExample:my_search_results.jsonExample:tidy_tweettree_search_2022-02-22.dbtree_search_2022-02-22.jsonLoading multiple JSON files into a databasetidy_tweet can accept more than one JSON file at a time. If you have multiple JSON files, for example resulting
from different search terms or Twitter accounts, you can list them all in a singletidy_tweetcommand:tidy_tweetDATABASEJSON_FILE_1JSON_FILE_2JSON_FILE_3For example:tidy_tweettree_searches_2022-02-22.dbpine_tree_2022-02-22.jsoneucalypt_2022-02-22.jsonjacaranda_2022-02-22.jsonAt present, there is no metadata to tell what data came from which file, but we plan to fix this soon!Python libraryHere is an example using the test data file included with tidy_tweet:fromtidy_tweetimportinitialise_sqlite,load_twarc_json_to_sqliteimportsqlite3initialise_sqlite('ObservatoryTeam.db')load_twarc_json_to_sqlite('tests/data/ObservatoryTeam.jsonl','ObservatoryTeam.db')withsqlite3.connect('ObservatoryTeam.db')asconnection:db=connection.cursor()db.execute("select count(*) from tweet")print(f"There are{db.fetchone()[0]}tweets in the database!")Feedback and contributionsWe appreciate all feedback and contributions!Found an issue with tidy_tweet?Find out how to let us knowInterested in contributing? Find out more in ourcontributing.mdAcknowledgementsSome of this documentation is copied fromGab Tidy Data,
and much of the structure and functionality is also modelled on gab_tidy_data, which was our initial foray into
developing a tool like this.About tidy_tweetTidy_tweet is created and maintained by theQUT Digital Observatoryand
is open-sourced under an MIT license. We welcome contributions and feedback!A DOI and citation information will be added in future.