Package: batchtools
batchtoolsPython-based Command Line Tools to interact with AWS BatchSetupFirst time installIt is suggested that you set up a virtual/conda environment forbatchtools.
Tools to interact with AWS require specific versions, which may conflict with other software on your system if not used in an isolated environment.Optional (but recommended): environment setupdownload and setup conda if you don't have it installed. See theirInstallation guidesetup environment:conda create -n batch python=3.6 pip
conda activate batchORuse a python virtual environmentInstall using pippip install batchtoolsDevelopmentclone this repo and cd into it:git clone git@gitlab.corticometrics.com:internal_projects/batchtools.git
cd batchtoolsinstall the package within your new environmentpip install -e .To install additional dev requirements (as well as interactive programming tools, likejupyterlabandipython), install the dev requirements from within thebatchconda environment:# make sure you're in this repo
cd /path/to/batchtools
# if not done already
conda activate batch
pip install -r requirements-dev.txtBefore any commit, run the toolblackto style code in a uniform format. This also includes anyversioneerrelated code (used to create the version of the tool, which is auto-generated)black --target-version py36 --exclude .*version.* .UsageCurrently supports two command line tools:submit_subjectsandcheck_status. Both require AWS Batch to be properly configured, and these commands need to be ran by AWS users with appropriate IAM permissions to access Batchsubmit_subjectsRun with--helpto see all options.An example command would be:submit_subjects \
  -q arn:aws:batch:us-east-1:123456789101:job-queue/my-queue \
  -j arn:aws:batch:us-east-1:123456789101:job-definition/my-definition:1 \
  -f /path/to/file_list \
  -o s3://my-bucket/path/to/output \
  -L /path/to/license_file.json  \
  -l /path/to/submission.jsonWhere/path/to/file_listis  file with one s3:// path per line, pointing to a dicom dir (a folder containing just a single T1w DICOM series to be processed)/path/to/license_file.jsonis a valid license provided by CorticoMetrics/path/to/submission.jsonis a JSON log file created here, with information about submitted subjectscheck_statusThis tool is still a work in progress, with more documentation coming soon! Run with--helpto see all informationTo continously check the status of subjects submitted above, run this command:check_status \
 -l /path/to/submission.json \
 -s /path/to/status.json \
 --poll \
 --continuousWhere/path/to/submission.jsonis a JSON log file created byubmit_subjects, with information about submitted subjects/path/to/status.jsonis a JSON log file created here, with status informationAfter submitted subjects have completed processing, outputs can be downloaded locally using a--get_*command.
For example, to download all output from re:THINQ (log, report.pdf, subject_info.json, and artifact.tar.gz) from subjects that both succeeded and failed in processing:check_status \
 --output_status both \
 -s /path/to/status.json \
 --save_location /path/to/results \
 --get_all \
 --ignore_nonexistantAWS Batch info (random notes)on AWS Batch, there's a Dashboard where you can see what is running. It shows status ofSUBMITTED,PENDING,STARTING,RUNNING,SUCCESSandFAILUREhttps://console.aws.amazon.com/batch/home?region=us-east-1#/dashboardwe can create a Job Queue to submit to (currently usingreTHINQ-testing), which defines which Compute Environment(s) to useThe Compute Environment defines instance type, and max number of CPUs to run at once
you submit a job based on a Job Definition, which goes to a specific queue, and can have specified commands, env variables, etc (we currently use variables so the job can find the correct subjectwithin the Job Definition, the Container image to use is defined, the amount of resources the job needs is specified, and any AWS IAM roles can be assignedas the job is running, you can click on it from within the dashboard, and there is a link to CloudWatch, where stdout/stderr are capturedstatus etc can be polled while the job is in the dashboard usingawscli(or usingboto3, the python SDK, from a script). this gives a json that will say its state,  a bunch of info on the job itself, and thelogStreamNameto find the job's logs in cloudwatchI haven't played around with this part too much (mostly refer to the dashboard for course success/failure messages), and i remember cloudwatch being annoying in the past to access with the cli, but i think that may have improved recentlyChanging the number of max CPUSTo change the number of CPUs and other features of the environment, go to the"Compute environments" section of AWS BATCH.Thec5.4xlargeis the recommended instance type.reTHINQrequires 16 vCPUs and 12 GB RAM to runSelect the environment, and click "Edit". Make sure the "Service role" is set toAWSBatchServiceRolebefore saving any changes.If the number number of CPUs gets too high, theEC2 Service Limitsmay need to be changed. Click on "Request limit increase" next to the instance type that you want to increase (default isc4.4xlarge), and fill out the "Service limit increase form". Be sure to select "Us East (Northern Virginia" as the region.kill all batch jobsaws batch list-jobs --job-queue reTHINQ-testing --job-status $STATUS > file.json # default is RUNNING, if you skip this flag, need to do RUNNABLE, etc to kill everything
# open test.json in an editor, select all the JobIDs
# JOBS is all the JobIDs
for i in $JOBS; do
  aws batch terminate-job --job-id $i --reason "whatever"
done