Package: gnip-historical
Python LibraryandCommand Line Utilitiesfor Gnip Historical PowerTrack APIThe process for launching and retrieveing data for an historical historical jobrequires only a few steps:1) create job2) retrieve and review job quote3) accept or reject job4) download data files list5) download dataUntilities are included to assist with each step.SETUP UTILITY=============First, set up your Gnip credentials.  There is a simple utility to create the local credentialfile named ".gnip".$ ./setup_gnip_creds.pyUsername: shendrickson@gnip.comPassword:Password again:Endpoint URL. Enter your Account Name (eg https://historical.gnip.com:443/accounts/<account name>/): shendricksonDone creating file ./.gnipBe sure to run:chmod og-w .gnip$ chmod og-w .gnipIf you use the example JSON job description, be sure to change the "serviceUserNameField"to your own, i.e., for Twitter, use your Twitter handle.You will likely wish to run these utilities from other directory locations so be sure the export anupdated PYTHONPATH,$ export PYTHONPATH=${PYTHONPATH}:path-to-gnip-python-historical-utilitiesCREATE JOB==========Create a job description by editing the example JSON file provided ("bieber_job1.json").You will end up with a single JSON record like this (see GNIP documentation for optiondetails). the fromDate and toDate are in the format YYYYmmddHHMM:{"dataFormat" : "activity-streams","fromDate" : "201201010000","publisher" : "twitter","rules" :[{"tag" : "bestRuleEver","value" : "bieber"}],"serviceUsername" : "PUT_YOUR_TWITTER_HANDLE_HERE","streamType" : "track","title" : "BieberJob1","toDate" : "201201010001"}To create the job,$ ./create_job.py -f./bieber_job1.json -t "Social Data Phenoms - Bieber"The response is the JSON record returned by the server. It will describe the job (includingJobID and the JobURL, or any error messages.To get help,$ ./create_job.py -hUsage: create_job.py [options]Options:-h, --help            show this help message and exit-u URL, --url=URL     Job url.-l, --prev-url        Use previous Job URL (only from this configurationfile.).-v, --verbose         Detailed output.-f FILENAME, --filename=FILENAMEFile defining job (JSON)-t TITLE, --title=TITLETitle of project, this title supercedes title in file.LIST JOBS, get JOB QUOTES and get JOB STATUS:=============================================$ ./list_jobs.py -hUsage: list_jobs.py [options]Options:-h, --help            show this help message and exit-u URL, --url=URL     Job url.-l, --prev-url        Use previous Job URL (only from this configurationfile.).-v, --verbose         Detailed output.-d SINCEDATESTRING, --since-date=SINCEDATESTRINGOnly list jobs after date, (default2012-01-01T00:00:00)For example, I have three completed jobs, a Gnip job, a Bieber job and a SXSWjob for which data is avaiable.$  ./list_jobs.py#########################TITLE:     GNIP2012STATUS:    finishedPROGRESS:  100.0 %JOB URL:   https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/eeh2vte64.json#########################TITLE:     Justin Bieber 2009STATUS:    finishedPROGRESS:  100.0 %JOB URL:   https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/j5epx4e5c3.json#########################TITLE:     SXSW2010-2012STATUS:    finishedPROGRESS:  100.0 %JOB URL:   https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/sbxff05b8d.jsonTo see detailed information or download data filelist,specify URL with -u or add -v flag (data_files.txt containsonly URLs from last job in list)DOWNLOAD URLS OF FILES CONTAINING DATA======================================To retrieve the file locations for the data files this job created on S3, passthe job URL with the -u flag (or if you used -u for this job previously, just use -l--see help),$  ./list_jobs.py -u  https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/sbxff05b8d.json#########################TITLE:     SXSW2010-2012STATUS:    finishedPROGRESS:  100.0 %JOB URL:   https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/sbxff05b8d.jsonRESULT:Job completed at ........ 2012-09-01 04:35:23No. of Activities ....... -1No. of Files ............ -1Files size (MB) ......... -1Data URL ................ https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historical/track/jobs/sbxff05b8d/results.jsonDATA SET:No. of URLs ............. 131,211File size (bytes)........ 2,151,308,466Files (URLs) ............ https://archive.replay.historicals.review.s3.amazonaws.com/historicals/twitter/track/activity-streams/shendrickson/2012/08/28/20100101-20120815_sbxff05b8d/2010/01/01/00/00_activities.json.gz?AWSAccessKeyId=AKIAJ7O2S22DN2NDN7UQ&Expires=1349066046&Signature=hDSc0a%2BRQeG%2BknaSAWpzSUoM1F0%3Dhttps://archive.replay.historicals.review.s3.amazonaws.com/historicals/twitter/track/activity-streams/shendrickson/2012/08/28/20100101-20120815_sbxff05b8d/2010/01/01/00/10_activities.json.gz?AWSAccessKeyId=AKIAJ7O2S22DN2NDN7UQ&Expires=1349066046&Signature=DOZlXKuMByv5uKgmw4QrCOpmEVw%3Dhttps://archive.replay.historicals.review.s3.amazonaws.com/historicals/twitter/track/activity-streams/shendrickson/2012/08/28/20100101-20120815_sbxff05b8d/2010/01/01/00/20_activities.json.gz?AWSAccessKeyId=AKIAJ7O2S22DN2NDN7UQ&Expires=1349066046&Signature=X4SFTxwM2X9Y7qwgKCwG6fH8h7w%3Dhttps://archive.replay.historicals.review.s3.amazonaws.com/historicals/twitter/track/activity-streams/shendrickson/2012/08/28/20100101-20120815_sbxff05b8d/2010/01/01/00/30_activities.json.gz?AWSAccessKeyId=AKIAJ7O2S22DN2NDN7UQ&Expires=1349066046&Signature=WVubKurX%2BAzYeZLX9UnBamSCrHg%3Dhttps://archive.replay.historicals.review.s3.amazonaws.com/historicals/twitter/track/activity-streams/shendrickson/2012/08/28/20100101-20120815_sbxff05b8d/2010/01/01/00/40_activities.json.gz?AWSAccessKeyId=AKIAJ7O2S22DN2NDN7UQ&Expires=1349066046&Signature=OG9ygKlXNxFvJLlAEWi3hes5yyw%3D...Writing files to data_files.txt...Filenames for the 131K files created on S3 by the job have been downloaded to a file inthe local directory, ./data_files.txt.DOWNLOAD DATA=============To retrieve this data use the utility,$ ./get_data_files.bash...This will lauch up to 8 simultaneousl cUrl connections to S3 to download the filesinto a local ./data/year/month/day/hour... directory tree (see name_mangle.py for details).ACCEPT/REJECT JOB=================After a job is quoted, you can accept or reject the job.  The job will not start until it is accepted.$ ./accept_job -u https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historicals/track/jobs/c9pe0day6h.jsonor$ ./reject_job -u https://historical.gnip.com:443/accounts/shendrickson/publishers/twitter/historicals/track/jobs/c9pe0day6h.jsonThe module gnip_historical.py provides additional functionality you can access programatically.==Gnip-Python-Historical-Utilities by Scott Hendrickson is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.