Package: twicorder-search
Twicorder SearchA Twitter crawler for Python 3 based on Twitter’s public API.Supported end pointsTwicorder Search currently supports the following end points:/1.1/followers/ids/1.1/friends/list/1.1/search/tweets/1.1/statuses/lookup/1.1/statuses/user_timeline/1.1/users/lookupTo add a new end point, fork the repository and add a new query type tosrc/twicorder/queries/request/endpoints. New endpoints should inherit fromBaseQueryor one of its derivatives and must implementname,endpointandresult_type.InstallationTwicorder Search can be installed using PIP:pipinstalltwicorder-searchFor a more comprehensive guide using a virtual environment, seeInstallation using Python 3 virtual environmentsRunning TwicorderAfter installing, there will be a new executable available,twicorder. Use this to run the
application:twicorder
Usage:twicorder[OPTIONS]COMMAND[ARGS]...TwicorderSearch

Options:--project-dirTEXTRootdirectoryforproject--helpShowthismessageandexit.

Commands:runStartcrawlerutilsUtilityfunctionsThe project dir is where Twicorder stores temporary files and logs. To specify a project directory
other than the default, use the flag--project-dir.:twicorder--project-dir/path/to/my_projectIf not provided, the current working directory is used.ConfigurationTwicorder can be configured by passing parameters in the command line interface or by setting
environment variables. The environment variables are laid out similar to their CLI counterparts.Specifying a task generator with CLItwicorderrun--task-genuser_timelineSpecifying a task generator with environment variableexportTWICORDER_RUN_TASK_GEN="user_timeline"Full list of CLI options:twicorderrun--help
Usage:twicorderrun[OPTIONS]Startcrawler

Options:--consumer-keyTEXTTwitterconsumerkey[required]--consumer-secretTEXTTwitterconsumersecret[required]--access-tokenTEXTTwitteraccesstoken[required]--access-secretTEXTTwitteraccesssecret[required]--out-dirTEXTCustomoutputdirforcrawleddata--out-extensionTEXTFileextensionforcrawledfiles(.txtor.zip)--task-fileTEXTYamlfilecontainingtaskstoexecute--full-user-mentionsFormentions,lookupfulluserdata--appdata-tokenTEXTAppdatatoken--user-lookup-intervalINTEGERMinutesbetweenlookupsofthesameuser[default:15]--appdata-timeoutFLOATSecondstotimeoutforinternaldatastore[default:5.0]--task-gen<TEXTTEXT>...Taskgenerator(s)touse.Example:"user_idname_pattern=/tmp/**/*_ids.txt,delimiter=,"[default:config]--remove-duplicatesEnsuresduplicatedtweets/usersarenotrecorded.Savesspace,butcanslowdownthecrawler.[default:True]--helpShowthismessageandexit.Task generatorsTwicorder can be configured with one or more task generators for creating API requests.Tasks fileThe tasks file is the default task generator for Twicorder and is used when no generator is
specified. By default Twocorder searches the project root for a file calledtasks.yaml.PROJECT_ROOT├──appdata│└──twicorder.sql├──logs│└──twicorder.log└──tasks.yamlIt is however possible to specify a different file path using--task-file:twicorder--task-file/path/to/my_file.yamlExample tasks.yaml fileUse this file to define the queries you wish to run and where to store their output data, relative
to the output directory. Frequency is given in minutes and defines how often a new scan will be
triggered for the given query.# Tasks## Queries are added on the form listed below.## free_search:                  # endpoint name#   - frequency: 60             # Interval between repeating queries in minutes#     output: github/mentions   # Output directory, relative to project directory#     kwargs:                   # Keyword Arguments to feed to endpoint#       q: @github              #   "q" for "query" in the case of free_search## See https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators# for how to use free search to find mentions, replies, hashtags etc.## See https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets# for keyword arguments to use with search.## See https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline# for keyword arguments to use with user timelines.user_timeline:-frequency:60output:"github/timeline"kwargs:screen_name:"github"-frequency:120output:"nasa/timeline"kwargs:screen_name:"NASA"free_search:-frequency:60output:"github/mentions"kwargs:q:"@github"-frequency:60output:"github/replies"kwargs:q:"to:github"-frequency:60output:"github/hashtags"kwargs:q:"#github"User LookupThe User Lookup generator takes one or more files with delimited user ids or
user names as input. It then generates tasks to fetch user objects for each id
or user name.twicorderrun--task-genuser_lookupsname_pattern=/taskgen/*.txt,lookup_method=usernameKeyword ArgumentTypeDescriptionname_patternstrPOSIX style search patterndelimiterstrDefault:"\n"lookup_methodstr"id"or"username"User TimelineThe User Timeline generator takes one or more files with delimited user ids or
user names as input. It then generates tasks to fetch tweets for each user's
timeline.twicorderrun--task-genuser_timelinename_pattern=/taskgen/*.txt,lookup_method=id,max_requests=5Keyword ArgumentTypeDescriptionname_patternstrPOSIX style search patterndelimiterstrDefault:"\n"lookup_methodstr"id"or"username"max_requestsintMax number of requests before the task is considered donemax_ageintMax age in days for a tweet before the query should be considered doneCreate new task generatorTwicorder supports creating custom task generators. To create a generator, create a class that inherits fromBaseTaskGeneratorand
implement thenameclass attribute and thefetch()method. Seetwicorder/tasks/generators/user_lookup_generator.pyfor an example.Place the custom task
generator in a suitable directory and point to said directory with the
environment variableTWICORDER_TASKGEN_PATH:exportTWICORDER_TASKGEN_PATH="/path/to/my/generator_dir"The task generator file name must end in_generator.py:$TWICORDER_TASKGEN_PATH├──__init__.py└──custom_task_generator.pyClearing temporary files or logsUse theutilscommand to clean up temporary files and logs:twicorderutils--help
Usage:twicorderutils[OPTIONS]Utilityfunctions

Options:--clear-cacheClearcacheandexit--purge-logsPurgelogsandexit--helpShowthismessageandexit.DockerDocker Compose ExamplesCrawl data based on entries in thetasks file.version:"3"services:twicorder-search:build:./image:twicorder-search:devrestart:unless-stoppedcontainer_name:twicorder-searchnetwork_mode:bridgeenvironment:-TWICORDER_RUN_CONSUMER_KEY=XXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_CONSUMER_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_ACCESS_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_ACCESS_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_REMOVE_DUPLICATES=0-TWICORDER_RUN_APPDATA_TOKEN=searchvolumes:-/home/user/project/data:/data-/home/user/project/config:/configCrawl tweets using theuser_timelinetask generator. The generator reads all *.txt files located
in/home/user/project/taskgenon the host system (name_pattern=/taskgen/*.txt) and expects to
find one user ID (lookup_method=id) per line. For each user the number of page results are limited
to 5 (max_requests=5).version:"3"services:twicorder-timeline:build:./image:twicorder-timeline:devrestart:on-failurecontainer_name:twicorder-timelinenetwork_mode:bridgeenvironment:-TWICORDER_RUN_CONSUMER_KEY=XXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_CONSUMER_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_ACCESS_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_ACCESS_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-TWICORDER_RUN_FULL_USER_MENTIONS=0-TWICORDER_RUN_REMOVE_DUPLICATES=0-TWICORDER_RUN_APPDATA_TOKEN=timeline-TWICORDER_RUN_TASK_GEN=user_timeline name_pattern=/taskgen/*.txt,lookup_method=id,max_requests=5volumes:-/home/user/project/data:/data-/home/user/project/taskgen:/taskgen