Package: twitterhistory
Download a history of posts and user metadata from the microblogging service TwitterThis is a Python module to download a complete history of posts and user metadata from the microblogging service Twitter using its API’s as of 2021 latest version 2. Data are saved to an SQLAlchemy/GeoAlchemy2-compatible database (currently only PostgreSQL/PostGIS is fully supported, see also thedocumention of GeoAlchemy2).The script will download all Twitter status messages up until the current time, and keep track of already downloaded time periods in a cache file (default location~/.cache/twitterhistory.yml). When started the next time, it will attempt to fill gaps in the downloaded data and catch up until the then current time.To usetwitterhistory, your API key (see further down) needs to be associated to an account withacademic research access.If you usetwitterhistoryfor academic research, please cite it in your publication:Fink, C. (2021):twitterhistory: a Python tool to download historical Twitter data.doi:10.5281/zenodo.4471195DependenciesThe script is written in Python 3 and depends on the Python modulesblessed,GeoAlchemy2,psycopg2,PyYaml,RequestsandSQLAlchemy.InstallationpipinstalltwitterhistoryConfigurationCopy the example configuration filetwitterhistory.yml.exampleto a suitable location, depending on your operating system:on Linux systems:system-wide configuration:/etc/twitterhistory.ymlper-user configuration:~/.config/twitterhistory.ymlOR${XDG_CONFIG_HOME}/twitterhistory.ymlon MacOS systems:per-user configuration:${XDG_CONFIG_HOME}/twitterhistory.ymlon Microsoft Windows systems:per-user configuration:%APPDATA%\twitterhistory.ymlin a custom file path location specified on the command line (see further down)Adapt the configuration:Configure a database connection string (connection_string), pointing to an existing database (with the PostGIS extension enabled).Configure an APIOAuth 2.0 Bearer tokenwith access to the Twitter API v2twitter_oauth2_bearer_token).Configure one or more search terms for the query (search_terms).If you have a cache file from a previous installation in which already downloaded time periods are saved, copy it to${XDG_CACHE_HOME}/twitterhistory.ymlor%LOCALAPPDATA%/twitterhistory.ymlon Linux or MacOS, and Microsoft Windows, respectively.The cache file is currently also the best way to limit the temporal range of the data collection (by default,twitterhistorydownloads the entire history of Tweets that correspond to the search terms). Runtwitterhistoryat least briefly for it to create an initial cache file. In this file, it marks the time spans for which it successfully downloaded data, persearch_term. Add one or more!TimeSpanobjects that cover all periods between March 2006 and the current date except the temporal range you want to download -twitterhistorywill then try to fill this gap, only.UsageCommand line executablepython-mtwitterhistorypython-mtwitterhistory--config/path/to/custom/config-file.ymlPythonImport thetwitterhistorymodule. Instantiate aTwitterHistoryDownloader, and call itsdownload()method.importtwitterhistorydownloader=twitterhistory.TwitterHistoryDownloader()downloader.download()Data privacyBy default,twitterhistorypseudonymises downloaded metadata, i.e., it replaces (direct) identifiers with randomised identifiers (generated using hashes, i.e., one-way ‘encryption’). This serves as one step of a responsible data processing workflow. However, other (meta-)data might nevertheless qualify as indirect identifiers, as they, combined or on their own, might allow re-identification of a person. If you want to use data downloaded usingtwitterhistoryin a GDPR-compliant fashion, you have to follow up the data collection stage with data minimisation and further pseudonymisation or anonymisation efforts.twitterhistorycan keep original identifiers (i.e., skip pseudonymisation). To instruct it to do so, instantiate aTwitterHistoryDownloaderwith the parameterpseudonymise_identifiers=Falseor set the according parameter in the configuration file. Ensure that you fulfil all legal and organisational requirements to handle personal information before you decide to collect non-pseudonyismed data.importtwitterhistorydownloader=twitterhistory.TwitterHistoryDownloader(pseudonymise_identifiers=False# get legal advice and ethics approval before doing this)downloader.download()