Package: dozent
DozentDozent is a powerful downloader that is used to collect large amounts of Twitter data from the internet archive.It is built on top ofPySmartDLand multithreading, similar to how traditional download accelerators likeaxel,aria2candaws s3work, ensuring that the biggest bottlenecks are your network and your hardware.The data that is downloaded is already heavily compressed to reduce download times and save local storage. When uncompressed, the data can easily add up to several terabytes depending on the timeframe of data being collected. Fortunately, you do not have to decompress the data to analyze it! We are working on a separate big data tool namedMurpheusthat uses Dask to analyze the data without needing to decompress it.If you have any ideas on how to improve Dozent, please open an issuehereand tell us how!InstallationBefore installing, ensure that the version of python that you're using is python>=3.6. We intend to support all of the latest releases of as they come outInstalling with pipInstalling with pip is as easy as:pipinstalldozentInstalling with DockerIn keeping with our goal for keeping everything we distribute as lightweight as possible, we include a docker image that would ensure that this process is as painless as possible without having to worry about python versions and so on.While "installing" isn't something that we can do with docker, we felt it best to include a some helpful links to help new comers install docker.You can find the link to the installationhere. If you chose to go this route, we suggest jumping down to theRun Dozent as a Docker Containersection after installing docker.Usage$python-mdozent--help

usage:__main__.py[-h]-sSTART_DATE-eEND_DATE[-tTIMEIT][-oOUTPUT_DIRECTORY][-q]Apowerfuldownloadertogettweetsfromtwitterforourcompute.Thefirst
stepofmany

optionalarguments:-h,--helpshowthishelpmessageandexit-sSTART_DATE,--start-dateSTART_DATEThedatefromwherewedownload.Theformatmustbe:YYYY-MM-DD-eEND_DATE,--end-dateEND_DATEThelastdaythatwedownload.Theformatmustbe:YYYY-MM-DD-tTIMEIT,--timeitTIMEITShowtotalprogramruntime-oOUTPUT_DIRECTORY,--output-directoryOUTPUT_DIRECTORYOutputDirectorywherethefilewillbestored.Defaultstothedata/directory-q,--quietTurnoffoutput(exceptforerrorsandwarnings)Downloading with Dozent after installing with pipDownloading all tweets from 12th of May 2020 to 15th of May 2020$python-mdozent-s2020-05-12-e2020-05-15

Queueingtweetsdownloadfor05-2020
Queueingtweetsdownloadfor05-2020
Queueingtweetsdownloadfor05-2020
Queueingtweetsdownloadfor05-2020
https://archive.org/download/archiveteam-twitter-stream-2020-05/twitter_stream_2020_05_13.tar[downloading]16Mb/2498Mb@1.6MB/s[------------------][0%,32minutes,31secondsleft]Downloading with Dozent after installing DockerPull the latest Dozent image from Docker Hub$dockerpullsocialmediapublicanalysis/dozent:latestGet help$dockerrun-itsocialmediapublicanalysis/dozent:latestor$dockerrun-itsocialmediapublicanalysis/dozent:latest-hDownload all tweets from March 12th, 2020 to March 15th, 2020$dockerrun-itsocialmediapublicanalysis/dozent:latestpython-mdozent-s2020-05-12-e2020-05-15About the DataOnly collects Tweets in the English languageTweets are stored in JSON formatEach day is a compressed file roughly 2.5 GB large or ~ 32 GB uncompressedEach tweet has accompanying metadata about the tweet and userSample DataInterested in seeing what the data that Dozent collects looks like?Check it out!https://dozent-tests.s3.amazonaws.com/sample_data.json