Package: simeon
simeonsimeonis a CLI tool to help with the processing of edx Research
data. It canlist,download, andsplitedX data packages. It
can alsopushthe output of thesplitsubcommand to both GCS and
BigQuery. It is heavily inspired by theedx2bigquerypackage. If
you’ve used that tool, you should be able to navigate the quirks that
may come with this one.Installing from pypipython3-mpipinstallsimeon# Or with geoippython3-mpipinstallsimeon[geoip]# Then invoke the CLI tool withsimeon--helpInstalling with git clone and pipgitclonegit@github.com:MIT-IR/simeon.gitcdsimeon&&python3-mvenvvenv&&sourcevenv/bin/activate&&python-mpipinstall.# Or with geoipcdsimeon&&python3-mvenvvenv&&sourcevenv/bin/activate&&python-mpipinstall.[geoip]# Then invoke the CLI tool withsimeon--helpUsing Dockerdockerrun--rm-itmitir/simeon:latestsimeon--helpDevelopinggitclonegit@github.com:MIT-IR/simeon.gitcdsimeon# Set up a virtual environment if you don't already have onpython3-mvenvvenvsourcevenv/bin/activate# pip install the package in an editable waypython3-mpipinstall-e.[test,geoip]# Invoke the executablesimeon--help# Run the teststox# Write code and tests and submit PR'sSetups and configurationssimeonis a glorified downloader and uploader set of scripts. Much
of the downloading and uploading that it does makes the assumptions that
you have your AWS credentials configured properly and that you’ve got a
service account file for GCP services available on your machine. If the
latter is missing, you may have to authenticate to GCP services through
the SDK. However, both we and Google recommend you not do that.Every downloaded file is decrypted either during the download process or
while it gets split by thesimeon splitcommand. So, this tool
assumes that you’ve installed and configuredgpgto be able to
decrypt files from edX.The following steps may be useful to someone just getting started with
the edX data package:Credentials from edXReach out to edX to get your data czar credentialsConfigure both AWS and gpg, so your credentials can access the S3
buckets and yourgpgkey can decrypt the files thereSetup a GCP projectCreate a GCP projectSet up a BigQuery workspaceCreate a GCS bucketCreate a service account and download the associated fileGive the service account Admin Role access to both the BigQuery
project and the GCS bucketIf the above steps are carried out successfully, then you should be able
to usesimeonwithout any issues.However, if you’ve taken care of the above steps but are still unable to
getsimeonto work, please open an issue.Further,simeoncan parse INI formatted configuration files. It, by
default, looks for files in the user’s home directory, or in the current
working directory of the running process. The base names that are
targeted when config files are looked up are:simeon.cfgor.simeon.cfgorsimeon.inior.simeon.ini. You can also
providesimeonwith a config file by using the global option--config-fileor-Cand giving it a path to the file with the
corresponding configurations.The following is a sample file content:# Default section for things like the organization whose data package is processed
# You can also set a default site as one of the following: edx, edge, patches[DEFAULT]site=edxorg=yourorganizationxclistings_file=/path/to/file/with/course_ids# Section related to Google Cloud (project, bucket, service account)[GCP]project=your-gcp-project-idbucket=your-gcs-bucketservice_account_file=/path/to/a/service_account_file.jsonwait_for_loads=Truegeo_table=your-gcp-project.geocode_latest.geoipyoutube_table=your-gcp-project.videos.youtubeyoutube_token=your-YouTube-API-token# Section related to the AWS credentials needed to download data from S3[AWS]aws_cred_file=~/.aws/credentialsprofile_name=defaultThe options in the config file(s) should match the optional arguments of
the CLI tool. For instance, the--service-account-file,--projectand--bucketoptions can be provided under theGCPsection of the config file asservice_account_file,projectandbucket, respectively. Similarly, the--siteand--orgoptions can be provided under theDEFAULTsection assiteandorg, respectively.List filessimeoncan list files on S3 for your organization based on criteria
like file type (sqlorlogoremail), time intervals (begin
and end dates), and site (edxoredgeorpatches).Example: List the latest data packages for file typessql,email, andlog# List the latest SQL bundlesimeonlist-sedx-omitx-fsql-L# List the latest email data dumpsimeonlist-sedx-omitx-femail-L# List the latest tracking log filesimeonlist-sedx-omitx-flog-LDownload and split filessimeoncan download, decrypt and split up files into folders
belonging to specific courses.Example 1: Download, split and push SQL bundles to both GCS and
BigQuery# Download the latest SQL bundlesimeondownload-sedx-omitx-fsql-L-ddata/# Download SQL bundles dumped any time since 2021-01-01 and
# extract the contents for course ID MITx/12.3x/1T2021.
# Place the downloaded files in data/ and the output of the split operation
# in data/SQLsimeondownload-sedx-omitx-c"MITx/12.3x/1T2021"-fsql\-b2021-01-01-ddata-S-Ddata/SQL/# Push to GCS the split up SQL files inside data/SQL/MITx__12_3x__1T2021simeonpushgcs-fsql-p${GCP_PROJECT_ID}-b${GCS_BUCKET}\-S${SAFILE}data/SQL/MITx__12_3x__1T2021# Push the files to BigQuery and wait for the jobs to finish
# Using -s or --use-storage tells BigQuery to extract the files
# to be loaded from Google Cloud Storage.
# So, use the option when you've already called simeon push gcssimeonpushbq-w-s-fsql-p${GCP_PROJECT_ID}-b${GCS_BUCKET}\-S${SAFILE}data/SQL/MITx__12_3x__1T2021Example 2: Download, split and push tracking logs to both GCS and
BigQuery# Download the latest tracking log filesimeondownload-sedx-omitx-flog-L-ddata/# Download tracking logs dumped any time since 2021-01-01
# and extract the contents for course ID MITx/12.3x/1T2021
# Place the downloaded files in data/ and the output of the split operation
# in data/TRACKING_LOGSsimeondownload-sedx-omitx-c"MITx/12.3x/1T2021"-flog\-b2021-01-01-ddata-S-Ddata/TRACKING_LOGS/# Push to GCS the split up tracking log files inside
# data/TRACKING_LOGS/MITx__12_3x__1T2021simeonpushgcs-flog-p${GCP_PROJECT_ID}-b${GCS_BUCKET}\-S${SAFILE}data/TRACKING_LOGS/MITx__12_3x__1T2021# Push the files to BigQuery and wait for the jobs to finish
# Using -s or --use-storage tells BigQuery to extract the files
# to be loaded from Google Cloud Storage.
# So, use the option when you've already called simeon push gcssimeonpushbq-w-s-flog-p${GCP_PROJECT_ID}-b${GCS_BUCKET}\-S${SAFILE}data/TRACKING_LOGS/MITx__12_3x__1T2021If you already have downloaded SQL bundles or tracking log files, you
can usesimeon splitthem up.Make secondary/aggregated tablessimeoncan generate secondary tables based on already loaded data.
Callsimeon report--helpfor the expected positional and optional
arguments.Example: Makeperson_coursefor course IDMITx/12.3x/1T2021# Make a person course table for course ID MITx/12.3x/1T2021
# Provide the -g option to give a geolocation BigQuery table
# to fill the ip-to-location details in the generated person course tableCOURSE=MITx/12.3x/1T2021simeonreport-w-g"${GCP_PROJECT_ID}.geocode.geoip"-t"person_course"\-p${GCP_PROJECT_ID}-S${SAFILE}${COURSE}Notes:Please note that SQL bundles are quite large when split up, so
consider using the-cor--coursesoption when invokingsimeon download-Sorsimeon splitto make sure that you
limit the splitting to a set of course IDs. You may also use the--clistings-fileoption, which expects a txt file of course IDs;
one ID per line. If the aforementioned options are not used,simeonmay end up failing to complete the split operation due to
exhausted system resources (storage to be specific).simeon downloadwith file typeslogandemailwill both
download and decrypt the files matching the given criteria. If the
latter operations are successful, then the encrypted files are
deleted by default. This is to make sure that you don’t exhaust
storage resources. If you wish to keep those files, you can always
use the--keep-encryptedoption that comes withsimeon downloadandsimeon split. SQL bundles are only
downloaded (not decrypted). Their decryption is done during asplitoperation.Unless there is an unhandled exception (which should be reported as a
bug),simeonshould, by default, print to the standard output
both information and errors encountered while processing your files.
You can capture those logs in a file by using the global option--log-fileand providing a destination file for the logs.When using multi argument options like--tablesor--courses,
you should try not to place them right before the expected positional
arguments. This will help the CLI parser not confuse your positional
arguments with table names (in the case of--tables) or course
IDs (when--coursesis used).Splitting tracking logs is a resource intensive process. The routine
that splits the logs generates a file for each course ID encountered.
If you happen to have more course IDs in your logs than the running
process can open operating system file descriptors, thensimeonwill put away records it can’t save to disk for a second pass.
Putting away the records involves using more memory than normally
required. The second pass will only require one file descriptor at a
time, so it should be safe in terms of file descriptor limits. To
helpsimeonnot have to do a second pass, you may increase the
file descriptor limits of processes from your shell by running
something likeulimit-n2000before callingsimeon spliton
Unix machines. For Windows users, you may have to dig into the
Windows Registries for a corresponding setting. This should tell your
OS kernel to allow OS processes to open up to 2000 file handles.Care must be taken when usingsimeon splitandsimeon pushto
make sure that the number of positional arguments passed does not
lead to the invoked command exceeding the maximum command-line length
allowed for arguments in a command. To avoid errors along those
lines, please consider passing the positional arguments as UNIX glob
patterns. For instance,simeon split--file-typelog'data/TRACKING-LOGS/*/*.log.gz'tellssimeonto expand the given glob pattern, instead of relying
on the shell to do it.Thereportsubcommand relies on the presence of SQL query files
to parse and send to BigQuery to execute. Any errors arising from
executing the parsed queries will be shown to the end user through
the given log stream. While thesimeontool ships with query
files for most secondary/reporting tables that are based on theedx2bigquerytool, an end user should be able to pointsimeonto a different location with SQL query files by using the--query-diroption that comes withsimeon report.
Additionally, these query files can containjinja2
templatedSQL code.
Any mentioned variables within these templated queries can be passed
tosimeon reportby using the--extra-argsoption and passing
key-value pair items in the formatvar1=value1,var2=value2,var3=value3,...,varn=valuen. Further,
these key-value pair items can also be typed by using the formatvar1:i=value1,var2:s=value2,var3:f=value3,...,varn:s=valuen. In
this format, the type is append to the key, separated by a colon. The
only supported scalar types, so far, aresforstr,iforint, andfforfloat. If any conversion errors occur
during value parsing, then those are shown to the end user, and the
query won’t get executed. Finally, if you wish to pass anarrayorlistto the template, you will need to repeat a key multiple
times. For instance, if you want to pass a list namedmylistcontaining the integers, you could write something like--extra-argsmylist:i=1,mylist:i=2,mylist:i=3. This means that
you’ll have a pythonlistnamedmylistwithin your template,
and it should contain[1, 2, 3].