Package: taupe
TaupeA simple program to extract the URLs of your tweets, retweets, replies, quote tweets, and "likes" from a personal Twitter archive.Table of contentsIntroductionInstallationUsageKnown issues and limitationsGetting helpContributingLicenseAcknowledgmentsIntroductionWhen youdownload your personal Twitter archive, you receive aZIPfile. The contents are not necessarily in a format convenient for doing something with them. For example, you may want to send the URLs to theWayback Machine at the Internet Archiveor do something else with the URLs. For tasks like that, you need to extract URLs from your Twitter archive. That's the purpose of Taupe.Taupe(a loose acronym ofTwitterarchiveURLparser) takes a Twitter archive ZIP file, extracts the URLs corresponding to your tweets, retweets, replies, quote tweets, and liked tweets, and outputs the results in acomma-separated values (CSV)format that you can easily use with other software tools. Once you haveinstalled it, usingtaupeis easy:# Extract tweets, retweets, replies, and quote tweets:taupe/path/to/your/twitter-archive.zip# Extract likes:taupe--extractlikes/path/to/your/twitter-archive.zip# Learn more:taupe--helpInstallationThere are multiple ways of installing Taupe.  Please choose the alternative that suits you.Alternative 1: installing Taupe usingpipxPipxlets you install Python programs in a way that isolates Python dependencies, and yet the resultingtaupecommand can be run from any shell and directory – like any normal program on your computer. If you usepipxon your system, you can install Taupe with the following command:pipxinstalltaupePipx can also let you run Taupe directly usingpipx run taupe, although in that case, you must always prefix every Taupe command withpipx run.  Consult thedocumentation forpipx runfor more information.Alternative 2: installing Taupe usingpipYou should be able to installtaupewithpipfor Python 3.  To installtaupefrom thePython package repository (PyPI), run the following command:python3-mpipinstalltaupeAs an alternative to getting it fromPyPI, you can usepipto installtaupedirectly from GitHub:python3-mpipinstallgit+https://github.com/mhucka/taupe.gitIf you already installed Taupe once before, and want to update to the latest version, add--upgradeto the end of either command line above.Alternative 3: installing Taupe from sourcesIf  you prefer to install Taupe directly from the source code, you can do that too. To get a copy of the files, you can clone the GitHub repository:gitclonehttps://github.com/mhucka/taupeAlternatively, you can download the software source files as a ZIP archive directly from your browser using this link:https://github.com/mhucka/taupe/archive/refs/heads/main.zipNext, after getting a copy of the files,  runsetup.pyinside the code directory:cdtaupe
python3setup.pyinstallUsageIf the installation process described above is successful, you should end up with a program namedtaupein a location where software is normally installed on your computer.  Runningtaupeshould be as simple as running any other command-line program. For example, the following command should print a helpful message to your terminal:taupe--helpIf not given the option--helpor--version, this program expects to be given apersonal Twitter archive file, either on the command line (as an argument) or on standard input (from a pipe or file redirection). Here's an example (and note this path is fake – substitute a real path on your computer when you do this!):taupe/path/to/twitter-archive.zipThe URLs produced bytaupewill be, by default, as they appear in the archive. If you want tonormalize the URLsinto the canonical formhttps://twitter.com/twitter/status/TWEETID, use the option--canonical-urls(-cfor short):taupe-c/path/to/twitter-archive.zipThe structure of the outputThe option--extractcontrols both the content and the format of the output. The following options are recognized:ValueSynonymOutputall-tweetstweetsCSV table with all tweets and details (default)my-tweetslist of URLs of only your original tweetsretweetslist of URLs of tweets that are retweetsquoted-tweetsquote-tweetslist of URLs of other tweets you quotedreplied-tweetsreply-tweetslist of URLs of other tweets you replied tolikedlikeslist of URLs of tweets you "liked"all-tweetsWhen using--extract all-tweets(the default),taupeproduces a table with four columns.  Each row of the table corresponds to a type of event in the Twitter timeline: a tweet, a retweet, a reply to another tweet, or a quote tweet. The values in the columns provide details about the event. The following is a summary of the structure:Column 1Column 2Column 3Column 4tweet timestamp in ISO formatThe URL of the tweetThe type; one oftweet,reply,retweet, orquote(For typereplyorquote.) The URL of the original or source tweetThe last column only has a value for replies and quote-tweets; in those cases, the URL in the column refers to the tweet being replied to or the tweet being quoted.  The fourth column does not have a value for retweets even though it would be desirable, because the Twitter archive – strangely – does not provide the URLs of retweeted tweets.Here is an example of the output:2022-09-21T22:36:29+00:00,https://twitter.com/mhucka/status/1572716422857658368,quote,https://twitter.com/poppy_northcutt/status/1572714310077673472
2022-10-10T22:04:20+00:00,https://twitter.com/mhucka/status/1579593701965582336,reply,https://twitter.com/arfon/status/1579572453726355456
2022-10-14T04:17:01+00:00,https://twitter.com/mhucka/status/1580774654217625600,tweet
2022-10-25T14:49:06+00:00,https://twitter.com/mhucka/status/1584919989307715586,retweet
...my-tweetsWhen using--extract my-tweets, the output is just a single column (a list) of URLs, one per line, of just your original tweets. This list corresponds exactly to column 2 in the--extract all-tweetscase above.retweetsWhen using--extract retweets, the output is a single column (a list) of URLs, one per line, of tweets that are retweets of other tweets. This list corresponds to the values of column 2 above when the type isretweet.Important: the Twitter archive does not contain the original tweet's URL, only the URL of your retweet. Consequently, the output for--extract retweetsisyourretweet's URL, not the URL of the source tweet.quoted-tweetsWhen using--extract quoted-tweets, the output is a list of the URLs of other tweets that you have quoted. It corresponds to the subset of column 4 values above when the type is "quote". Note that these are the source tweet URLs, not the URLs of your tweets.replied-tweetsWhen using--extract replied-tweets, the output is a list of the URLs of other tweets that you have replied to. It corresponds to the subset of column 4 values above when the type is "reply". Note that these are the source tweet URLs, not the URLs of your tweets.likesWhen using the option--extract likes, the output will only contain one column: the URLs of the "liked" tweets.taupecannot provide more detail because the Twitter archive format does not contain date/time information for "likes". (This is also why "likes" arenotpart of the output when--extract all-tweetsis used – there is no possible value for column 1.)Here is an example of the output when using--extract likesin combination with--canonical-urls:https://twitter.com/twitter/status/1588146224376463365
https://twitter.com/twitter/status/1588349144803905536
https://twitter.com/twitter/status/1590475356976578560
...Other options recognized bytaupeRunningtaupewith the option--helpwill make it print help text and exit without doing anything else.The option--outputcontrols wheretaupewrites the output. If the value given to--outputis-(a single dash), the output is written to the terminal (stdout). Otherwise, the value must be a file.If given the--versionoption, this program will print its version and other information, and exit without doing anything else.If given the--debugargument,taupewill output a detailed trace of what it is doing. The debug trace will be sent to the given destination, which can be-to indicate console output, or a file path to send the debug output to a file.Summary of command-line optionsThe following table summarizes all the command line options available.ShortLong form optMeaningDefault-c--canonical-urlsNormalize Twitter URLsLeave as-is-h--helpPrint help info and exit-eE--extractEExtract URL typeEall-tweets⚑-oO--outputOWrite output to fileOTerminal✦-V--versionPrint program version & exit-@OUT--debugOUTWrite debug output toOUT⚐⚑   Recognized values:all-tweets,tweets,my-tweets,retweets,quoted-tweets,replied-tweets, andlikes. Seesection abovefor more information.✦   To write to the console, you can also use the character-as the value ofO; otherwise,Omust be the name of a file where the output should be written.⚐   To write to the console, use the character-as the value ofOUT; otherwise,OUTmust be the name of a file where the output should be written.Known issues and limitationsThis program assumes that the Twitter archive ZIP file is in the format which Twitter produced in mid-November 2022. Twitter probably used a different format in the past, and may change the format again in the future, sotaupemay or may not work on Twitter archives obtained in different historical periods.The Twitter archive format for "likes" contains only the tweet identifier and the text of the tweet; consequently,taupecannot provide date/time information for this case.This program does all its work in memory, which means thattaupe's ability to process a given archive depends on its size and how much RAM the computer has. It has only been tested with modest-sized archives. It is unknown how it will behave with exceptionally large archives.Getting helpIf you find a problem or have a request or suggestion, please submit it inthe GitHub issue trackerfor this repository.ContributingI would be happy to receive your help and participation if you are interested.  Everyone is asked to read and respect thecode of conductwhen participating in this project.  Please feel free toreport issuesor do apull requestto fix bugs or add new features.LicenseThis software is Copyright (C) 2022, by Michael Hucka.  This software is freely distributed under the MIT license.  Please see theLICENSEfile for more information.AcknowledgmentsThis work is a personal project developed by the author, using computing equipment owned by theCalifornia Institute of Technology Library.Thevector artworkof a bird, used as the icon for this repository, was created byNoe Araujofrom the Noun Project.  It is licensed under the Creative CommonsCC-BY 3.0license. I manually changed the color to be a shade of taupe.Taupe uses multiple other open-source packages, without which it would have taken much longer to write the software. I want to acknowledge this debt. In alphabetical order, the packages are:Aenum– Python package for advanced enumerationsCommonPy– a collection of commonly-useful Python functionsPlac– a command line argument parserRich– library for writing styled text to the terminalSidetrack– simple debug logging/tracing packageTwine– utilities for publishing Python packages onPyPI