Package: twilight
twilightTools for accessing theTwitter API
v1.1with paranoid
timeouts and de-pagination.Node.jstwitter-curlfor querying the streaming API (/sample.jsonand/filter.json).rtcountfor pulling out the retweets from a stream of JSON
tweets, and counting them.Pythonjson2ttv2converts directories full of Twitter.jsonfiles
into.ttv2files, bzip2’ing them, and ensuring that the result is
within a reasonable size of the source (greater than 2%, but less
than 6%) before deleting the original json.twitter-userpulls down the ~3,200 (max) tweets that are
accessible for a given user (also depends on the~/.twitterauth
file).Crawling quickstartInstall fromnpm:npm install -g twilightOr github (to make sure you’re getting the most up-to-date version):npm install -g git://github.com/chbrown/twilightAuthenticateThis app uses only OAuth 1.0A, which is mandatory. As of June 11, 2013,basic HTTP authentication is
disabledin the Twitter
Streaming API. So get some OAuth credentials togetherreal
quickand make a csv file that
looks like this:consumer_keyconsumer_secretaccess_tokenaccess_token_secretziurk0An7…VKmTsGrk2JjH…91505165…VcLOIzA0mkiCSbU…63Yp9EG4t…DhrlIQBMUaoL…91401882…XJa4HQKMgqfd7ee……………Theremustbe a header line withexactlythe following values:consumer_keyconsumer_secretaccess_tokenaccess_token_secretTab / space seperated is fine, and any other columns will simply be
ignored, e.g., if you want to record thescreen_nameof each
account. Also, order doesn’t matter – your headers just have to line up
with their values.Thetwitter-curlscript expects to find this file at~/.twitter,
but you can specify a different path with the--accountscommand
line argument.From my/etc/supervisor/conf.d/*(Seehttp://supervisord.org/to get that all set up.)[program:justinnnnnn]user=chbrowncommand=twitter-curl--filter"track=loveyabiebs,belieber,bietastrophe"--file/data/twitter/justin_TIMESTAMP.json--timeout86400--interval3600--ttv2twitter-curloptions--accountsshould point to a file with OAuth Twitter account
credentials. Currently, the script will simply use a random row from
this file.--filtercan be anytrack=whateverorlocations=-18,14,68,44etc. Aquerystring-parsable string. If
no filter is specified, it will use the spritzer at/sample.json--fileshouldn’t require creating any directions, and the
TIMESTAMP bit will be replaced by a filesystem-friendly iso
representation of whenever the program is started.// Specifically:varstamp=newDate().toISOString().replace(/:/g,'-').replace(/\..+/,'');// stamp == '2013-06-07T15-47-49'--timeout(seconds) the program will die with error code 1 after
this amount of time. Don’t specify a timeout if you don’t want this.--interval(seconds) the amount of time to allow for silence from
Twitter before dying. Also exits with code 1. Defaults to 600 (10
minutes).--ttv2(boolean) output TTV2 normalized flat tweets instead of
full JSON.Because in most cases of error the script simply dies, this approach
only really makes sense if you’re putting it behind some process
monitor. (By the way, I’ve tried most of them: monitd, daemontools’s
svc, god, bluepill, node-forever—and supervisord is by far the best.)The script does not abide by any Twitter backoff requirement, but I’ve
never had any trouble with backoff being enforced by Twitter. It’s more
than curl, though, because it checks that it’s receiving data. Often,
withcurlandPycURL, my connection would be dropped by Twitter,
but no end signal would be sent. My crawler would simply hang, expecting
data, but would not try to reconnect.But beyond that, without--ttv2, it doesn’t provide anything more
thancurl.TTV2TTV2 is the Tweet tab-separated format version 2, the specification is
below. Fields are 1-indexed for easy AWKing (see Markdown source for
0-indexing).tweet_idcreated_at parsed into YYYYMMDDTHHMMSS, implicitly UTCtext, newlines and tabs converted to spaces, html entities replaced,
t.co urls resolvedlon,latplace_idplace_strin_reply_to_status_idin_reply_to_screen_nameretweet_id id of the original tweetretweet_countuser.screen_nameuser.iduser.created_at parsed into YYYYMMDDTHHMMSSuser.nameuser.descriptionuser.locationuser.urluser.statuses_countuser.followers_countuser.friends_countuser.favourites_countuser.geo_enableduser.default_profileuser.time_zoneuser.languser.utc_offsetThis format is not the default, and will be the output only when you use
the--ttv2option.ExamplesInstalljsonfirst:npm install json. It’s awesome.twitter-curl --filter 'track=bootstrap' | json -C text
twitter-curl --filter 'track=bootstrap' | json -C user.screen_name text
twitter-curl --filter 'track=انتخابات' | json -C text
twitter-curl --filter 'track=sarcmark,%F0%9F%91%8F' | json -C textIt supports unicode: انتخابات is Arabic for “elections,” anddecodeURIComponent('%F0%9F%91%8F')is the“CLAPPING HANDS”
(U+1F44F)character.If you use a filter with url-escaped characters in supervisord, note
that supervisord Python-interpolates strings, so you’ll need to escape
the percent signs, e.g.:[program:slowclap]
command=twitter-curl --filter "track=%%F0%%9F%%91%%8F" --file /tmp/slowclap.jsonTTV2 ExampleInstead of JSON, you can use AWK to look at the TTV2:twitter-curl --filter 'track=data,science' --ttv2 | awk 'BEGIN{FS="\t"}{print $4,$3}'StatsRSS usage per-process is between 20-40MB.VSZ on a machine running six of these crawlers is 80-90MB.Python contents vs. Javascript contentseasy_install -U twilightThe Python and Javascript components are mostly complementary. The
Javascript offers crawlers, Python provides post-processing.Testing with Travis CIThe tested CLI commands now check for OAuth in specific environment
variables before reading the given--accountsfile or the default
one (~/.twitter).To get tests to run on Travis CI, we can usetraviscommand line
tool to encrypt a quad of valid Twitter OAuth credentials so that only
Travis CI can see them.Put together a file that looks like this (call ittwilight.env):consumer_key=bepLTQD5ftZCjqhXgkuJW
consumer_secret=jZ4HEYgNRKwJykbh5ptmcqV7v0o2WODdiMTF1fl6B9X
access_token=167246169-e1XTUxZqLnRaEyBF8KwOJtbID26gifMpAjukN5vz
access_token_secret=OVm7fJt8oY0C9kBsvych6Duq5pNIUxwagG143HdRAnd then, from within the root directory of this git repository, run the
following sequence:gem install travis
travis encrypt -s -a < twilight.env.travis.ymlshould now have those variables, but encrypted with
Travis CI’s public key.LicenseCopyright © 2011–2013 Christopher Brown.MIT
Licensed.