Package: gdelttools
Loading GDELT data into MongoDBThis is a set of programs for loading theGDELT 2.0data set into MongoDB.Quick StartInstall the latest version of Python frompython.org.
You need at least version 3.6 for this program. Many versions of Python that
come pre-installed are only 2.7. This version will not work.Now installgdelttoolspipinstallgdelttoolsNow get the master file of all the GDELT files.gdeltloader--masterThis will generate a file named something likemasterfilelist.txtDownloading the master data setTo download the master data set associated with GDELT (the export files) you can combine
these steps:gdeltloader--master--download--overwriteThis will get the master file, parse it, extract the list of CSV files and unzip them.
the full GDELT 2.0 database runs to several terabytes of data so this is not recommend.Theoverwriteargument ruthlessly overwrites all files with extreme prejudice. Without
it thegdeltloaderscript will attempt to reuse the files you have already downloaded. As
each file is unique this may save time if you need to re-download some files.To limit the amount you download you can specify--lastto define how many files worth of data
you want to download:gdeltloader--master--download--overwrite--last20This command will download the most recent 20 files worth of data. Note that a file is a triplet ofexport,mentionsandgkgdata. If you only want one you should specify a--filter. Without the filter a command like the above will actually download 60 files.GDELT 2.0 Encoding and StructureTheGDELTdataset is a large dataset of news events that is updated
in real-time. GDELT stands for Global Database of Events Location and Tone. The format
of records in a GDELT data is defined by theGDELT 2.0 CodebookEach record uses an encoding method called CAMEO coding which is defined by theCAMEO Codebook.Once you understand the GDELT recording structure and the CAMEO encoding you will be able
to decode a record. To fully decode a record you may need theTABARIdictionaries
from which the CAMEO encoding is derived.How to download GDELT 2.0 dataThegdeltloaderscript can download cameo data and unzip the files so that
they can be loaded into MongoDB.usage: gdeltloader [-h] [--master] [--update] [--database DATABASE] [--collection COLLECTION]
  [--local LOCAL] [--overwrite] [--download] [--metadata]
  [--filter {all,gkg,mentions,export}] [--last LAST] [--version]

options:
  -h, --help            show this help message and exit
  --master              GDELT master file [False]
  --update              GDELT update file [False]
  --database DATABASE   Default database for loading [GDELT]
  --collection COLLECTION
                        Default collection for loading [events_csv]
  --local LOCAL         load data from local list of zips
  --overwrite           Overwrite files when they exist already
  --download            download zip files from master or local file
  --metadata            grab meta data files
  --filter {all,gkg,mentions,export}
                        download a subset of the data, the default is all data [export, mentions gkg, all]
  --last LAST           how many recent files to download default : [0] implies all files
  --version             show program's version number and exit

Version: 0.07b1 More info : https://github.com/jdrumgoole/gdelttoolsHere is how to download the last 5 hours of GDELT data.gdeltloader--master--update--download--last20This command will only download theexportfiles for the last 20 15-minute blocks, which
are the files we are interested in.How to import downloaded data into MongoDBNow import the CSV files withmongoimport.There is amongoimport.shscript in thegdelttoolsrepo
which is already configured with the right arguments.  There is also a corresponding
field file,gdelt_field_file.ffwhich this script uses to ensure correct type mappings.To run:shmongoimport.sh--uri"<YOUR-MONGODB-CONNECTION-STRING>"This will upload all the CSV files in the current working directory.