Package: twclient
twclientThis package provides a high-level command-line client for the Twitter API,
with a focus on loading data into a database for analysis or bulk use.Documentation:mit-ccc.github.io/twclientWhy use this project?This project offers high-level primitives for researchers who want to get
data out of Twitter, without worrying about the API details. The client can
handle multiple sets of API credentials seamlessly, helping avoid rate limit
issues.[1]There’s also support for exporting bulk datasets from the fetched
raw data.InstallationInstall the package from pypi:pip3installtwclientor, if you want to use the development version, clone this repo and install:gitclonegit@github.com:mit-ccc/twclient.git&&cdtwclientpip3install.You can also use the-eflag to install in editable mode:pip3install-e.To install all development dependencies, replace.with.[dev]in the
arguments topip3 install.UsageFirst, you need to tell twclient about your database backend and Twitter
credentials. On the database side, we’ve only tested with Postgres and SQLite.
While the package may well work with other DB engines, be aware that you may
encounter issues.Setup: DatabaseThe database backend can be either sqlite or an arbitrary database
specified by a sqlalchemy connection string.You can set up the database in one of two ways. Both create a persistent
profile in your.twclientrcfile (or whatever other file you specify), so
there’s no need to type the database details repeatedly.First, you can specify the DB with a sqlalchemy connection URL:# Postgres -- this becomes the default DB because you've created it firsttwclientconfigadd-db-u"postgresql+psycopg2://username@hostname:5432/dbname"my_postgres_db# Or you could use SQLitetwclientconfigadd-db-u"sqlite:///home/user/twitter.db"my_sqlite_dbThere’s also support for using SQLite without having to think about sqlalchemy
and connection URLs:twclientconfigadd-db-f./twitter2.dbmy_sqlite_db2If you specify a file-backed sqlite DB, as in the examples above, it’ll be
created if it doesn’t exist. Other databases (Postgres, for example) will need
to be set up separately.Finally, you have to install our database schema into your database to receive
Twitter data:# You have to specify the -y to say you're aware all data will be droppedtwclientinitialize-dmy_postgres_db-yBe aware that doing this willDROP ALL EXISTING TWCLIENT DATA!!! (Or other
tables with the same names.) If you’re not just getting started, check to make
sure you’re using a new or empty database, don’t care about the contents,
and/or have backups before running this.Setup: TwitterYou’ll also need to set up your Twitter API credentials.[1]As with the
database setup, doing this stores the credentials in a config file (the same
config file as for database info) for ease of use. Only two sets of credentials
are shown, but you can add as many as you want.Here’s an example of adding two API keys:twclientconfigadd-api-ntwitter1\--consumer-keyXXXXX\--consumer-secretXXXXXX\--tokenXXXXXX\--token-secretXXXXXXtwclientconfigadd-api-ntwitter2\--consumer-keyXXXXX\--consumer-secretXXXXXX\--tokenXXXXXX\--token-secretXXXXXXHere’s an example of adding credentials that useapp-only auth:twclientconfigadd-api-ntwitter3\--consumer-keyXXXXX\--consumer-secretXXXXXXPulling dataTo actually pull data, use thetwclient fetchcommand. We’ll pull
information about three specific users and a Twitter list here. Note that you
can refer to lists either by their “slug” (username/listname) or by the ID at
the end of a URL of the formhttps://twitter.com/i/lists/53603015.First, let’s load some users and their basic info:# you could instead also end this with "-l 53603015"; it's the same listtwclientfetchusers-nwwbrannonCCCatMITMIT-lMIT/peers1Now, to save typing, let’s use thetwclient tagcommand to apply a tag we
can use to keep track of these users later:twclienttagcreatesubjectstwclienttagapplysubjects-nwwbrannonCCCatMITMIT-lMIT/peers1We can now use this tag in specifying users, such as which users we’d like to
fetch tweets for:twclientfetchtweets-gsubjectsAnd if we also want their follow-graph info (note that a “friend” is Twitter’s
term for a follow-ee, an account you follow):twclientfetchfriends-gsubjectstwclientfetchfollowers-gsubjectsAt this point, the loaded data is in the database configured withconfigadd-db. Useful features have been normalized out to save processing time. The
raw API responses are also saved for later analysis.Exporting dataYou can query the data with the usual database tools (psqlfor postgres,sqlite3for sqlite, ODBC clients, etc.) or export certain pre-defined bulk
datasets with thetwclient exportcommand. For example, here are the follow
graph and mention graph over users:twclientexportfollow-graph-ofollow-graph.csvtwclientexportmention-graph-omention-graph.csvIf you want to restrict the export to only the users specified above:twclientexportfollow-graph-gsubjects-ofollow-graph.csvtwclientexportmention-graph-gsubjects-omention-graph.csvFor other exports and other options, see the documentation.Feedback or ContributionsIf you come across a bug, please report it on the Github issue tracker. If you
want to contribute, reach out! Extensions and improvements are welcome.CopyrightCopyright © 2019-2023 Massachusetts Institute of Technology.Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this software except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.[1](1,2)Of course, you’ll need to make sure you have the right to use all of
your credentials and are complying with Twitter’s terms of use.ChangelogAll notable changes to this project will be documented here.The format is based onKeep a Changelog, and this project followsSemantic
Versioning.v0.2.0AddedNew features:First stable version.API documentation.A command-line interface for easy automated fetching of data.A stable relational data model, to make further analysis or data processing
independent of the details of data ingest.Support for fetching follow-graph edges, tweets, lists and user info.Support for a variety of database backends via sqlalchemy.Type-2 SCD for tracking the follow graph and list membership over time.Many sanity checks in the data model to ensure correctness of loaded data.Loaded data is extensively normalized: mentions, replies, retweets,
and entities like hashtags are extracted into first-class objects for
more convenient and accessible analysis.Users can be tagged into arbitrary groups for greater convenience in
analysis or data collection.Support for both app-only and user authentication to the Twitter API.