Package: twcompose
TwComposeCLI to manage rules and start tweets collection from the Twitter Stream APIWith TwCompose, you can:Add, modify and delete Twitter stream rules in a simple configuration fileValidate that your rules are properly format before applying your changesGet volume estimation for your rules to stay within the rate limitsStart collecting tweets in the background (Docker) with error handling and restart mechanismInstallationInstalling TwCompose requires at least Python 3.8pipinstalltwcomposeUsageCreate a credentials fileFirst, we need to specify theTwitter authentication tokento connect to the Twitter Stream API.
This needs to be specified in a YAML file (calledcredentials.ymlby default) with the following format:twitter_token: "<TWITTER_BEARER_TOKEN>"Create a Twitter-Compose fileThe following is an example of atwitter-compose.ymlfile.
It defines stream parameters and rules as well as output driver to save collected tweets.# twitter-compose.yml
image_tag: "0.1.0"

output:
  driver: local
  path: ./data/
  options:
    max_file_size: 1048576

parameters:
  tweet_fields:
    - text

streams:
  cop26:
    - tag: COP26GDA
      value: "#COP26GDA"
    - tag: bare cop26
      value: cop26 OR COP26 OR Cop26Collection image referenceControls the name and version of the Docker image used for the collector container.# twitter-compose.yml
image_tag: "0.1.0"
image_name: "ghcr.io/smassonnet/twcollect"Output driver referenceControls how the collected tweets are being saved.
Only support saving to a local folder in gzip compressed JSONLines files.
Files are split according themax_file_sizeoption.# twitter-compose.yml
output:
  driver: local
  path: ./data/
  options:
    max_file_size: 1048576driverOnly supports collection to alocalfolder.pathPath to the local folder to save into.optionsmax_file_size(number of bytes): Tweets are written to a new file when the file size reaches that limit. Defaults to 1 Gb.Stream parameters referenceControls the fields collected from the tweets.# twitter-compose.yml
parameters:
  tweet_fields:
    - textSee theTwitter stream API referencefor documentation.Note that the following fields correspond to the Twitter fields ending with.fieldsinstead of_fields:media_fields:media.fieldsplace_fields:place.fieldspoll_fields:poll.fieldstweet_fields:tweet.fieldsuser_fields:user.fieldsStream rules referenceDefines the scope of tweet to collect. SeeTwitter stream rules for reference.It is organised as a mapping between a stream group name (cop26is the example below) and a list of Twitter stream rules.
Naming the stream rules with unique and comprehensive tags is highly recommended.# twitter-compose.yml
streams:
  cop26:
    - tag: COP26GDA
      value: "#COP26GDA"
    - tag: bare cop26
      value: cop26 OR COP26 OR Cop26Command-line intefaceRuntwitter-compose --helpfrom the command-line:usage: twitter-compose [-h] [-f TC_FILE] [-p PROJECT_NAME]
                       [--credentials-file CREDENTIALS]
                       [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       {config,up,status,stop,volume} ...

Manage Twitter streams

positional arguments:
  {config,up,status,stop,volume}
    config              Show parsed configuration
    up                  Update Twitter streams
    status              Status of defined streams
    stop                Stop Twitter streams
    volume              Estimation of the monthly volume of streams

optional arguments:
  -h, --help            show this help message and exit
  -f TC_FILE, --file TC_FILE
                        The file name of the twitter-compose configuration
  -p PROJECT_NAME, --project-name PROJECT_NAME
                        Name of the current project
  --credentials-file CREDENTIALS, -c CREDENTIALS
                        A yaml file with mapping between credential name and
                        value
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Logging levelconfigValidates and prints the parsedtwitter-compose.ymlconfiguration.upUpdate twitter stream rules and starts/updates the local running stream collector Docker container.
If takes an optional--checkargument to display the changes without running the update.statusShow the installed Twitter stream rules and the status of the stream collector.stopStop the Docker container running the collection.NoteThis project has been set up using PyScaffold 4.3.1. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.