Package: simple-sagemaker
Simple SagemakerAsimplerandcheaperway to distribute work (python/shell/training) work on machines of your choice in the (AWS) cloud.Blog posts:A quick introductionA detailed distributed pytorch model training exampleRequirementsPython 3.6+An AWS account + region and credentials configured for boto3, as explained on theBoto3 docs(Optional) TheDocker Engine, to be able to customize a docker image(Optional) TheDocker Compose, for local testingGetting startedTo installSimple Sagemakerpip install simple-sagemakerThen, to get the shell commandcat /proc/cpuinfo && nvidia-smirun on a single ml.p3.2xlarge instance, run the followingssmcommand (documentation of thessmCLI is givenbelow):ssmshell-pssm-ex-tshell-task-o./output--cmd_line"cat /proc/cpuinfo && nvidia-smi"Output including the logs with script stdout is downloaded to./output.$cat./output/logs/logs0
processor:0modelname:Intel(R)Xeon(R)CPUE5-2686v4@2.30GHz
cpucores:4....
processor:2....
processor:7....
+-----------------------------------------------------------------------------+|NVIDIA-SMI440.33.01DriverVersion:440.33.01CUDAVersion:10.2||-------------------------------+----------------------+----------------------+|GPUNamePersistence-M|Bus-IdDisp.A|VolatileUncorr.ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.||===============================+======================+======================||0TeslaV100-SXM2...On|00000000:00:1E.0Off|0||N/A46CP027W/300W|0MiB/16160MiB|0%Default|+-------------------------------+----------------------+----------------------+
....Similarly, to run the followingworker1.pyon twoml.p3.2xlargespotinstancesimporttorchforiinrange(torch.cuda.device_count()):print(f"-***- Device{i}:{torch.cuda.get_device_properties(i)}")Just run the belowssmcommand:ssmrun-pssm-ex-tpython-task-eworker1.py-o./output/example1--itml.p3.2xlarge--ic2The output is saved to./output/example1, logs to./output/example1/logs/logs0and./output/example1/logs/logs1:$cat./output/example1/logs/logs0
...
-***-Device0:_CudaDeviceProperties(name='Tesla V100-SXM2-16GB',major=7,minor=0,total_memory=16160MB,multi_processor_count=80)...It's recommended to review thefully featured advanced example, as a demonstration of most features.More examples (below)CLI based examples:Passing command line argumentsTask state and outputProviding input dataChaining tasksConfiguring the docker imageDefining code dependenciesAPI based example:Single file exampleMotivationWell, I couldn't find a simple and cheap way to run my existing code on cloud. And, I wanted to try having my own OS project including documentation and a full automated testing and publishing piepline, so here it is :)
Please refer tothis blog post.BackgroundSimple Sagemakeris a thin wrapper around SageMaker's training and processingjobs, that makes distribution of work (python/shell) onany supported instance typevery simple.The distribution solution is composed of two parts, one on each side: arunneron the client machine that manages the distribution process, and aworkerwhich is the code being distributed on the cloud.Therunneris the main part of this package, can mostly be controlled by using thessmcommand line interface (CLI), or be fully customized by using the python API.Theworkeris basically the work (shell/python code) being distributed. Python code may be adapted to use a smalltask_toolkitlibrary (that is automatically injected to theworker) for getting the environment configuration (WorkerConfig, seebelow), i.e. input/output/state paths, running parameters. Shell command can access the same parameters on the command line, and completion is determined by the exit code (i.e. 0 is a success) etc.Therunneris used to configuretasksandprojects:Ataskis a logical step that runs on a defined input and provide output. It's defined by providing a local code path, entrypoint, and a list of additional local dependenciesA SageMakerjobis ataskinstance, i.e. a singlejobis created each time ataskis executedState is maintained between consecutive execution of the sametask(see morebelow)If ataskwas completed, by returnin 0 retcode from all instances, it'll be skipped automatically on the next time (unless enforced otherwise)Aprojectis a series of relatedtasks, with possible dependenciesThe output of a completed task can be consumed as input by a consecutive taskMain features"Simpler" - Except for holding an AWS account credentials, no other pre-configuration nor knowledge is assumed (well, almost :). Behind the scenes you get:Jobs IAM role creation, including policies for accessing needed S3 bucketsBuilding and uploading a customized docker image to AWS (ECS service)Synchronizing local source code / input data to a S3 bucketDownloading the results from S3..."Cheaper" -"pay only for what you use", and saveup to 90% of the costwith spot instances, which got used by default!Abstraction of how data is maintained on AWS (S3 service)No need to mess with S3 paths, the data is automaticallyState is automatically maintained between consecutive execution ofjobsthat belongs to the sametaskA simple way to define how data flows betweentasksof the sameproject, e.g. how the firsttask's outputs is used as an input for a secondtask(Almost) no code changes are to the existing code - the API is mostly wrapped by a command line interface (namedssm) to control the execution (a.k.a implement therunner, see below)In most cases it's only about 1 line for getting the environment configuration (e.g. input/output/state paths and running parameters, seebelow) and passing it on to the original codeEasy customization of the docker image (based on a pre-built one)The rest of the SageMaker advantages, which (mostly) behaves "normally" as defined by AWS, e.g.(Amazon SageMaker Developer Guide)[https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html](Amazon SageMaker Python SDK @ Read the Docs)[https://sagemaker.readthedocs.io/en/stable/index.html]High level flow diagramWorker environmentThe worker entry point (entry_pointparameter), directory (source_dirfor python code / .sh script,dir_filesfor shell script),
along with all dependencies (dependenciesparameter) are getting copied to a single directory (/opt/ml/code) on each instance,
and the entry point is then executed.
On top of the above, for python code tasks, thetask_toolkitlibrary is also added as a dependency in this folder.ConfigurationThe worker can access the environment configuration parameters in two ways:The environment variables, e.g.SM_NUM_CPUSrepresents the number of CPUs.Using theworker_liblibrary: initialize aWorkerConfiginstance,worker_config = worker_lib.WorkerConfig(), and then all params can be accessible from theworker_configvariable, e.g.worker_lib.num_cpusis the number of CPUs.The complete list of configuration parameters:DescriptionEnvironment variableworker_configfield nameExampleThe name of the current runningjobSAGEMAKER_JOB_NAMEjob_name'task1-2020-09-23-17-12-46-0JNcrR6H'Input channels:Names of the input channelsSM_CHANNELSchannels['data']The data input channelSM_CHANNEL_DATAchannel_data'/opt/ml/input/data/data'Path where the input model (given bymodel_uriparameter) is locatedSM_CHANNEL_MODELchannel_model'/opt/ml/input/data/model'Generally - path where the channel [ChannelName] is locatedSM_CHANNEL_[ChannelName]channel_[ChannelName]'/opt/ml/input/data/[ChannelName]'Additional command line parameters / hyperparametersSM_HPShps{'arg': 'hello world!', 'task': 1, 'worker': 1}State:The root path of where state should be storedSSM_STATEstate'/state'The instance specific state pathSSM_INSTANCE_STATEinstance_state'/state/algo-1'Output:The path where output data should be storedSM_OUTPUT_DATA_DIRoutput_data_dir'/opt/ml/output/data'Path where model output should be storedSM_MODEL_DIRmodel_dir'/opt/ml/model'TensorBoard outputSSM_TENSORBOARD_DIRtensorboard_dir'/opt/ml/output/tensorboard'System:The number of available CPUs on this instanceSM_NUM_CPUSnum_cpus2The number of available GPUs  instanceSM_NUM_GPUSnum_gpus1Name of the current hostSM_CURRENT_HOSTcurrent_host'algo-1'Names of all other hosts that are running on thisjobSM_HOSTShosts['algo-1', 'algo-2']The name of the network interfaceSM_NETWORK_INTERFACE_NAMEnetwork_interface_name'eth0'The number of instance running for thisjobSSM_NUM_NODESnum_nodes'eth0'The rank of the current instanceSSM_HOST_RANKhost_rank'eth0'StateState is maintained between executions of the sametask, i.e. betweenjobsthat belongs to the sametask.
The local path is available inworker_config.state.
When running multiple instances, the state data is merged into a single directory (post execution).  To avoid collisions, set theper_instance_stateparameter ofWorkerConfigconstructor toTrue(the default behavior), which initializes a per instance sub directory, and keep it inworker_config.instance_state. On top of that, the return value plays an important part: returning 0 means thejobis completed. If all instances of ajobmarked it as completed, thetaskis assumed to be completed by thatjob, which allows:To skip it next time (unless enforced otherwise by using--force_runningor a newer run of the sametaskfailed)To use its output as input for othertasks(see below:"Chaining tasks")OutputOn top of the state, there're 3 main other output mechanisms:Logs - any output written to standard output / errorOutput data - any data inworker_config.output_data_diris compressed into a output.tar.gz. Only the main instance output data is kept.Model - any data inworker_config.model_diris compressed into a model.tar.gz. As data from all instance is merged, be carful with collisions.Data maintenance on S3All data, including input, code, state and output, is maintained on S3. The bucket to use can be defined, or the default one is used.
The files and directories structure is as follows:[Bucket name]/[Project name]/[Task name]
|-- state
|-- input
|-- [Job name]
|   |-- output
|   |   |-- model.tar.gz
|   |   `-- output.tar.gz
|   `-- source/sourcedir.tar.gz
|-- [Job name 2]
|        ...state - the task state, shared between all jobs, i.e. task executionsinput - the task input, shared as well[Job name] - a per job specific foldermodel.tar.gz - model output data, merged fromall instancesoutput.tar.gz - themain instanceoutput data (other outputs are ignored)sourcedir.tar.gz - source code and dependencies[Job name 2] - another execution of the same taskLocal modeSageMaker offers partial"local mode"support in order to test locally. The basic mode runs just the docker locally, while keep using S3 for input/output, and there's thelocal_codemode that does everything locally.
To use the basic mode withSimple Sagemaker,localorlocal_gpuas instance type andlocal_mode = TrueforSageMakerProjectconstructor (this is done automatically withssmCLI).
Notes:sagemaker[local] is required:pip install sagemaker[local]Local mode doesn't support all features, e.g. state isn't supported. More notes and exclusions can be seen on thedocumentationlocal_codemode isn't currently supported bySimple SagemakerDistributed trainingSagemaker's PyTorch and TensorFlow pre-built images has extra customization for distributed training. Make sure to configureframework,framework_versionandpy_versionto use the image that matches your needs (the full list is here). For TensorFlow you'll need to use thedistributionparameters. For more details on the built in support see:PyTorch -Distributed PyTorch TrainingTensorFlow -Distributed TensorFlow Training.A full distributed ImageNet training pipeline can be foundhere, along with a detailed explanation on theblog post,Processing tasksDocumentation TBD. For now, take a lookon the processing cli examples, and thessm process -houtput.CLIThessmCLI supports 4 commands:run - to run a python / .sh script based taskshell - to run a shell based taskdata - to manage (download/clear state) the data of an existing taskprocess - to run a processing command, script or generic$ssm-h
usage:ssm[-h]{run,shell,data,process}...

positionalarguments:{run,shell,data,process}runRunapython/.shscripttaskshellRunashelltaskdataManagetaskdataprocessRunaprocessingtask

optionalarguments:-h,--helpshowthishelpmessageandexitssm run$ssmrun-h
usage:ssmrun[-h]--project_namePROJECT_NAME[--prefixPREFIX]--task_nameTASK_NAME[--bucket_nameBUCKET_NAME][--source_dirSOURCE_DIR]--entry_pointENTRY_POINT[--dependenciesDEPENDENCIES[DEPENDENCIES...]][--instance_typeINSTANCE_TYPE][--instance_countINSTANCE_COUNT][--volume_sizeVOLUME_SIZE][--no_spot][--use_spot_instances][--max_wait_minsMAX_WAIT_MINS][--max_run_minsMAX_RUN_MINS][--aws_repo_nameAWS_REPO_NAME][--repo_nameREPO_NAME][--image_tagIMAGE_TAG][--docker_file_path_or_contentDOCKER_FILE_PATH_OR_CONTENT][--framework{pytorch,tensorflow}][--framework_versionFRAMEWORK_VERSION][--py_versionPY_VERSION][--input_pathINPUT_PATH[INPUT_PATH...]][--model_uriMODEL_URI][--input_s3INPUT_S3[INPUT_S3...]][--input_taskINPUT_TASK[INPUT_TASK...]][--force_running][--distributionDISTRIBUTION][--clean_state][--keep_state][--metric_definitionsnameregexp][--tagkeyvalue][--output_pathOUTPUT_PATH][--download_state][--download_model][--download_output]optionalarguments:-h,--helpshowthishelpmessageandexit--project_namePROJECT_NAME,-pPROJECT_NAMEProjectname.(default:None)--prefixPREFIXS3prefix.(default:None)--task_nameTASK_NAME,-tTASK_NAMETaskname.(default:None)--bucket_nameBUCKET_NAME,-bBUCKET_NAMES3bucketname(adefaultoneisusedifnotgiven).(default:None)Code:--source_dirSOURCE_DIR,-sSOURCE_DIRPath(absolute,relativeoranS3URI)toadirectorywithanyothersourcecodedependenciesasidefromtheentrypointfile.Ifsource_dirisanS3URI,itmustpointtoatar.gzfile.StructurewithinthisdirectoryarepreservedwhenrunningonAmazonSageMaker.(default:None)--entry_pointENTRY_POINT,-eENTRY_POINTPath(absoluteorrelative)tothelocalPythonsourcefileora.shscriptwhichshouldbeexecutedastheentrypoint.Ifsource_dirisspecified,thenentry_pointmustpointtoafilelocatedattherootofsource_dir.(default:None)--dependenciesDEPENDENCIES[DEPENDENCIES...],-dDEPENDENCIES[DEPENDENCIES...]Alistofpathstodirectories(absoluteorrelative)withanyadditionallibrariesthatwillbeexportedtothecontainerThelibraryfolderswillbecopiedtoSageMakerinthesamefolderwheretheentrypointiscopied.(default:None)Instance:--instance_typeINSTANCE_TYPE,--itINSTANCE_TYPETypeofEC2instancetouse.(default:ml.m5.large)--instance_countINSTANCE_COUNT,--icINSTANCE_COUNTNumberofEC2instancestouse.(default:1)--volume_sizeVOLUME_SIZE,-vVOLUME_SIZESizeinGBoftheEBSvolumetouseforstoringinputdata.Mustbelargeenoughtostoreinputdata.(default:30)--no_spotUseondemandinstances(default:True)--use_spot_instancesSpecifieswhethertouseSageMakerManagedSpotinstances.(default:True)--max_wait_minsMAX_WAIT_MINSTimeoutinminuteswaitingforspotinstances.AfterthisamountoftimeAmazonSageMakerwillstopwaitingforSpotinstancestobecomeavailable.If0isspecifiedandspotinstancesareused,itssettomax_run_mins(default:0)--max_run_minsMAX_RUN_MINSTimeoutinminutesforrunning.AfterthisamountoftimeAmazonSageMakerterminatesthejobregardlessofitscurrentstatus.(default:1440)Image:--aws_repo_nameAWS_REPO_NAME,--arAWS_REPO_NAMENameofECSrepository.(default:None)--repo_nameREPO_NAME,--rnREPO_NAMENameoflocalrepository.(default:None)--image_tagIMAGE_TAGImagetag.(default:latest)--docker_file_path_or_contentDOCKER_FILE_PATH_OR_CONTENT,--dfDOCKER_FILE_PATH_OR_CONTENTEitherapathtoadirectorycontainingtheDockerFileoritscontent.Inthefirstcase,thebaseimageshouldbesetto`__BASE_IMAGE__`withintheDockerfile,andisautomaticallyreplacedwiththecorrectbaseimage.Forthelattercase,itshouldbethecontentofthedockerfilee.g.'--df "RUN pip3install pandas==0.25.3 scikit-learn==0.21.3"'(default:None)--framework{pytorch,tensorflow},-f{pytorch,tensorflow}Theframeworktouse,seehttps://github.com/aws/deep-learning-containers/blob/master/available_images.md(default:pytorch)--framework_versionFRAMEWORK_VERSION,--fvFRAMEWORK_VERSIONTheframeworkversion(default:None)--py_versionPY_VERSION,--pvPY_VERSIONThepythonversion(default:None)Running:--force_running,--frForcerunningthetaskevenifitsalreadycompleted.(default:False)--distributionDISTRIBUTIONTensorflowsdistributionpolicy,seehttps://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/using_tf.html#distributed-training.(default:None)--tagkeyvalueTagtobeattachedtothejobsexecutedforthistask.(default:None)I/O:--input_pathINPUT_PATH[INPUT_PATH...],-iINPUT_PATH[INPUT_PATH...]INPUT:PATH[DISTRIBUTION][SUBDIR]Local/s3pathfortheinputdata.Ifalocalpathisgiven,itwillbesyncedtothetaskfolderontheselectedS3bucketbeforelaunchingthetask.(default:None)--model_uriMODEL_URIURIwhereapre-trainedmodelisstored,eitherlocallyorinS3.Ifspecified,theestimatorwillcreateachannelpointingtothemodelsothetrainingjobcandownloadit.Thismodelcanbea‘model.tar.gz’fromaprevioustrainingjob,orotherartifactscomingfromadifferentsource.(default:None)--input_s3INPUT_S3[INPUT_S3...],--iisINPUT_S3[INPUT_S3...]INPUT_S3:INPUT_NAMES3_URI[DISTRIBUTION][SUBDIR]AdditionalS3inputsources(afewcanbegiven).(default:None)--input_taskINPUT_TASK[INPUT_TASK...],--iitINPUT_TASK[INPUT_TASK...]INPUT_TASK:INPUT_NAMETASK_NAMETYPE[DISTRIBUTION][SUBDIR]Useanoutputofacompletedtaskinthesameprojectasaninputsource(afewcanbegiven).Typeshouldbeoneof['state','model','source','output'].(default:None)--clean_state,--csClearthetaskstatebeforerunningit.Thetaskwillberunningagainevenifitwasalreadycompletedbefore.(default:False)--keep_state,--ksKeepthecurrenttaskstate.Ifthetaskisalreadycompleted,itscurrentoutputwillbetakenwithoutrunningitagain.(default:True)--metric_definitionsnameregexp,--mdnameregexpNameandregexpforametricdefinition,afewcanbegiven.Seehttps://docs.aws.amazon.com/sagemaker/latest/dg/training-metrics.html.(default:None)Download:--output_pathOUTPUT_PATH,-oOUTPUT_PATHLocalpathtodownloadtheoutputsto.(default:None)--download_stateDownloadthestateoncetaskisfinished(default:False)--download_modelDownloadthemodeloncetaskisfinished(default:False)--download_outputDownloadtheoutputoncetaskisfinished(default:False)Anythingafter"--"(followedbyaspace)willbepassedas-istotheexecuted
scriptcommandlinessm process$ssmprocess-h
usage:ssmprocess[-h]--project_namePROJECT_NAME[--prefixPREFIX]--task_nameTASK_NAME[--bucket_nameBUCKET_NAME][--codeCODE][--entrypointENTRYPOINT[ENTRYPOINT...]][--dependenciesDEPENDENCIES[DEPENDENCIES...]][--commandCOMMAND[COMMAND...]][--instance_typeINSTANCE_TYPE][--instance_countINSTANCE_COUNT][--volume_sizeVOLUME_SIZE][--max_run_minsMAX_RUN_MINS][--aws_repo_nameAWS_REPO_NAME][--repo_nameREPO_NAME][--image_tagIMAGE_TAG][--docker_file_path_or_contentDOCKER_FILE_PATH_OR_CONTENT][--frameworkFRAMEWORK][--framework_versionFRAMEWORK_VERSION][--input_pathINPUT_PATH[INPUT_PATH...]][--input_s3INPUT_S3[INPUT_S3...]][--input_taskINPUT_TASK[INPUT_TASK...]][--force_running][--clean_state][--keep_state][--tagkeyvalue][--envkeyvalue][--argumentsARGUMENTS[ARGUMENTS...]][--output_pathOUTPUT_PATH][--download_state][--download_model][--download_output]optionalarguments:-h,--helpshowthishelpmessageandexit--project_namePROJECT_NAME,-pPROJECT_NAMEProjectname.(default:None)--prefixPREFIXS3prefix.(default:None)--task_nameTASK_NAME,-tTASK_NAMETaskname.(default:None)--bucket_nameBUCKET_NAME,-bBUCKET_NAMES3bucketname(adefaultoneisusedifnotgiven).(default:None)Code:--codeCODEAnS3URIoralocalpathtoafilewiththeframeworkscripttorun.(default:None)--entrypointENTRYPOINT[ENTRYPOINT...],-eENTRYPOINT[ENTRYPOINT...]Theentrypointfortheprocessingjob(default:None).Thisisintheformofalistofstringsthatmakeacommand(default:None)--dependenciesDEPENDENCIES[DEPENDENCIES...],-dDEPENDENCIES[DEPENDENCIES...]Alistofpathstodirectories(absoluteorrelative)withanyadditionallibrariesthatwillbeexportedtothecontainerThelibraryfolderswillbecopiedtoSageMakerinthesamefolderwheretheentrypointiscopied.(default:None)--commandCOMMAND[COMMAND...]Thecommandtorun,alongwithanycommand-lineflags(defaultsto:"python3").(default:None)Instance:--instance_typeINSTANCE_TYPE,--itINSTANCE_TYPETypeofEC2instancetouse.(default:ml.t3.medium)--instance_countINSTANCE_COUNT,--icINSTANCE_COUNTNumberofEC2instancestouse.(default:1)--volume_sizeVOLUME_SIZE,-vVOLUME_SIZESizeinGBoftheEBSvolumetouseforstoringinputdata.Mustbelargeenoughtostoreinputdata.(default:30)--max_run_minsMAX_RUN_MINSTimeoutinminutesforrunning.AfterthisamountoftimeAmazonSageMakerterminatesthejobregardlessofitscurrentstatus.(default:1440)Image:--aws_repo_nameAWS_REPO_NAME,--arAWS_REPO_NAMENameofECSrepository.(default:None)--repo_nameREPO_NAME,--rnREPO_NAMENameoflocalrepository.(default:None)--image_tagIMAGE_TAGImagetag.(default:latest)--docker_file_path_or_contentDOCKER_FILE_PATH_OR_CONTENT,--dfDOCKER_FILE_PATH_OR_CONTENTEitherapathtoadirectorycontainingtheDockerFileoritscontent.Inthefirstcase,thebaseimageshouldbesetto`__BASE_IMAGE__`withintheDockerfile,andisautomaticallyreplacedwiththecorrectbaseimage.Forthelattercase,itshouldbethecontentofthedockerfilee.g.'--df "RUN pip3install pandas==0.25.3 scikit-learn==0.21.3"'(default:None)--frameworkFRAMEWORK,-fFRAMEWORKTheframeworktouse,seehttps://github.com/aws/deep-learning-containers/blob/master/available_images.md(default:sklearn)--framework_versionFRAMEWORK_VERSION,--fvFRAMEWORK_VERSIONTheframeworkversion(default:0.20.0)Running:--force_running,--frForcerunningthetaskevenifitsalreadycompleted.(default:False)--tagkeyvalueTagtobeattachedtothejobsexecutedforthistask.(default:None)--envkeyvalueEnvironmentvariablesfortherunningtask.(default:None)--argumentsARGUMENTS[ARGUMENTS...]Alistofstringargumentstobepassedtoaprocessingjob.Argumentscanalsobeprovidedafter"--"(followedbyaspace),whichmaybeneededforparameterswithdashes(default:None)I/O:--input_pathINPUT_PATH[INPUT_PATH...],-iINPUT_PATH[INPUT_PATH...]INPUT:PATH[DISTRIBUTION][SUBDIR]Local/s3pathfortheinputdata.Ifalocalpathisgiven,itwillbesyncedtothetaskfolderontheselectedS3bucketbeforelaunchingthetask.(default:None)--input_s3INPUT_S3[INPUT_S3...],--iisINPUT_S3[INPUT_S3...]INPUT_S3:INPUT_NAMES3_URI[DISTRIBUTION][SUBDIR]AdditionalS3inputsources(afewcanbegiven).(default:None)--input_taskINPUT_TASK[INPUT_TASK...],--iitINPUT_TASK[INPUT_TASK...]INPUT_TASK:INPUT_NAMETASK_NAMETYPE[DISTRIBUTION][SUBDIR]Useanoutputofacompletedtaskinthesameprojectasaninputsource(afewcanbegiven).Typeshouldbeoneof['state','model','source','output'].(default:None)--clean_state,--csClearthetaskstatebeforerunningit.Thetaskwillberunningagainevenifitwasalreadycompletedbefore.(default:False)--keep_state,--ksKeepthecurrenttaskstate.Ifthetaskisalreadycompleted,itscurrentoutputwillbetakenwithoutrunningitagain.(default:True)Download:--output_pathOUTPUT_PATH,-oOUTPUT_PATHLocalpathtodownloadtheoutputsto.(default:None)--download_stateDownloadthestateoncetaskisfinished(default:False)--download_modelDownloadthemodeloncetaskisfinished(default:False)--download_outputDownloadtheoutputoncetaskisfinished(default:False)Anythingafter"--"(followedbyaspace)willbepassedas-istotheexecuted
scriptcommandlineRunning a shell based task is very similar, except forsource_dirandentry_pointwhich are replaced bydir_filesandcmd_line, respectively. Runssm shell -hfor more details.ssm dataTo manage the data of an existing command:$ssmdata-husage:ssmdata[-h]--project_namePROJECT_NAME--task_nameTASK_NAME[--bucket_nameBUCKET_NAME][--output_pathOUTPUT_PATH][--download_state][--download_model][--download_output]optionalarguments:-h,--helpshowthishelpmessageandexit--project_namePROJECT_NAME,-pPROJECT_NAMEProjectname.--task_nameTASK_NAME,-tTASK_NAMETaskname.--bucket_nameBUCKET_NAME,-bBUCKET_NAMES3bucketname(adefaultoneisusedifnotgiven).--output_pathOUTPUT_PATH,-oOUTPUT_PATHLocalpathtodownloadtheoutputsto.--download_stateDownloadthestateoncetaskisfinished--download_modelDownloadthemodeloncetaskisfinished--download_outputDownloadtheoutputoncetaskisfinishedA fully featured advanced exampleAnd now to a real advanced and fully featured version, yet simple to implement.
In order to exemplify most of the possible features, the following files are used inCLI Example 6_1:.
|-- Dockerfile
|-- code
|   |-- internal_dependency
|   |   `-- lib2.py
|   |-- requirements.txt
|   `-- worker6.py
|-- data
|   |-- sample_data1.txt
|   `-- sample_data2.txt
`-- external_dependency
    `-- lib1.pyDockerfile - the dockerfile specifying how to extend the pre-built image# __BASE_IMAGE__ is automatically replaced with the correct base imageFROM__BASE_IMAGE__RUNpip3installpandas==0.25.3scikit-learn==0.21.3code - the source code folderinternal_dependency - a dependency that is part of the source code folderrequirements.txt - pip requirements file lists needed packages to be installed before running the workertransformers==3.0.2worker6.py - the worker codedata - input data filesexternal_dependency - additional code dependencyThe code is then launched a few time byrun.sh, to demonstrate different features:# Example 6_1 - a complete example part 1.#   - Uses local data folder as input, that is distributed among instances (--i, ShardedByS3Key)#   - Uses a public s3 bucket as an additional input (--iis)#   - Builds a custom docker image (--df, --repo_name, --aws_repo_name)#   - Hyperparameter task_type#   - 2 instance (--ic)#   - Use an on-demand instance (--no_spot)ssmrun-pssm-ex-ttask6-1-s$BASEDIR/example6/code-eworker6.py\-i$BASEDIR/example6/dataShardedByS3Key\--iispersonss3://awsglue-datasets/examples/us-legislators/all/persons.json\--df$BASEDIR/example6--repo_name"task6_repo"--aws_repo_name"task6_repo"--no_spot\--ic2--task_type1-o$1/example6_1# Example 6_2 - a complete example part 2.#   - Uses outputs from part 1 (--iit)#   - Uses additional local code dependencies (-d)#   - Uses the tensorflow framework as pre-built image (-f)#   - Tags the jobs (--tag)#   - Defines sagemaker metrics (--md)ssmrun-pssm-ex-ttask6-2-s$BASEDIR/example6/code-eworker6.py\-d$BASEDIR/example6/external_dependency--iittask_6_1_modeltask6-1model\--iittask_6_1_statetask6-1stateShardedByS3Key\-ftensorflow--md"Score""Score=(.*?);"--tag"MyTag""MyValue"\--ic2--task_type2-o$1/example6_2&# Running task6_1 again#   - A completed task isn't executed again, but the current output is used instead.#       --ks (keep state, the default) is used to keep the current statessmrun-pssm-ex-ttask6-1-s$BASEDIR/example6/code-eworker6.py\-i$BASEDIR/example6/dataShardedByS3Key\--iispersonss3://awsglue-datasets/examples/us-legislators/all/persons.json\--df$BASEDIR/example6--repo_name"task6_repo"--aws_repo_name"task6_repo"\--ic2--task_type1-o$1/example6_1>$1/example6_1_2_stdout--ks&wait# wait for all processesworker6.pycontains the following:logger.info("Score=10;")time.sleep(60)# sleep to be able to see the two scoreslogger.info("Score=20;")This get captured by the"Score=(.*?);"regular expression in thessmcommand above, then the metrics graphs can be viewed on the AWS console:More information can be foundhere.Feel free to dive more into thefiles of this example. Specifically, note how thesame worker codeis used for the two parts, and thetask_typehyperparameter is used to distinguish between the two.More examplesCLI based examples:A fully featured advanced examplePassing command line argumentsTask state and outputProviding input dataChaining tasksConfiguring the docker imageDefining code dependenciesAPI based example:Single file examplePassing command line argumentsAny extra argument passed to the command line in the form of --[KEY_NAME] [VALUE] is passed as an hyperparameter, and anything after "--" (followed by a space) in passed as-is to the executed script command line. hyperparameters are accessible for theworkerby thehpsdictionary within the environment configuration or just by parsing the command time argument of the running script (e.g. sys.argv).
For example, see the following worker codeworker2.py:fromworker_toolkitimportworker_libworker_config=worker_lib.WorkerConfig(False)print("-***-",worker_config.hps["msg"])Runner CLI:ssmrun-pssm-ex-ttask2-eworker2.py--msg"Hello, world!"-o./output/example2Output from the log fileInvoking script with the following command:

/opt/conda/bin/python worker2.py --msg Hello, world!

Hello, world!Task state and outputA complete example can be seen inworker3.py:importosfromworker_toolkitimportworker_libworker_config=worker_lib.WorkerConfig(False)open(os.path.join(worker_config.output_data_dir,"output_data_dir"),"wt").write("output_data_dir file")open(os.path.join(worker_config.model_dir,"model_dir"),"wt").write("model_dir file")open(os.path.join(worker_config.state,"state_dir"),"wt").write("state_dir file")# 0 retcode - marks the tasks as completed, to allow other tasks to use its output,# and to avoid re-running it (unless enforced)Runner CLI:ssmrun-pssm-ex-ttask3-eworker3.py-o./output/example3Output from the log fileInvoking script with the following command:

/opt/conda/bin/python worker2.py --msg Hello, world!

Hello, world!Providing input dataAJobcan be configured to get a few data channels:A single local path can be used with the-i/--input_pathargument. This path is synchronized to thetaskdirectory on the S3 bucket before running thetask. On theworkerside the data is accessible inworker_config.channel_dataAdditional S3 paths (many) can be set as well. Each input source is provided with--iis [name] [S3 URI], and is accessible by the worker withworker_config.channel_[name]when [name] is the same one as was provided on the command line.Setting an output of a anothertaskon the sameproject, see below"Chaining tasks"Assuming a localdatafolder containing a singlesample_data.txtfile, a complete example can be seen inworker4.py:importloggingimportsubprocessimportsysfromworker_toolkitimportworker_liblogger=logging.getLogger(__name__)deflistDir(path):logger.info(f"*** START listing files in{path}")logger.info(subprocess.run(["ls","-la","-R",path],stdout=subprocess.PIPE,universal_newlines=True).stdout)logger.info(f"*** END file listing{path}")if__name__=="__main__":logging.basicConfig(stream=sys.stdout)worker_config=worker_lib.WorkerConfig(False)listDir(worker_config.channel_data)listDir(worker_config.channel_bucket)Running command:ssmrun-pssm-ex-ttask4-eworker4.py-i./data\--iisbuckets3://awsglue-datasets/examples/us-legislators/all/persons.json-o./output/example4Output from the log file...
INFO:__main__:*** START listing files in /opt/ml/input/data/data
INFO:__main__:/opt/ml/input/data/data:
total 12
drwxr-xr-x 2 root root 4096 Sep 14 21:51 .
drwxr-xr-x 4 root root 4096 Sep 14 21:51 ..
-rw-r--r-- 1 root root   19 Sep 14 21:51 sample_data.txt

INFO:__main__:*** END file listing /opt/ml/input/data/data
INFO:__main__:*** START listing files in /opt/ml/input/data/bucket
INFO:__main__:/opt/ml/input/data/bucket:
total 7796
drwxr-xr-x 2 root root    4096 Sep 14 21:51 .
drwxr-xr-x 4 root root    4096 Sep 14 21:51 ..
-rw-r--r-- 1 root root 7973806 Sep 14 21:51 persons.json

INFO:__main__:*** END file listing /opt/ml/input/data/bucket
...Chaining tasksThe output of a completedtaskon the sameprojectcan be used as an input to anothertask, by using the--iit [name] [task name] [output type]command line parameter, where:[name] - is the name of the input source, accessible by the worker withworker_config.channel_[name][task name] - the name of thetaskwhose output is used as input[output type] - thetaskoutput type, one of "model", "output", "state"Using the model output oftask3and the sameworker4.pycode, we can now run:ssmrun-pssm-ex-ttask5-eworker4.py--iitbuckettask3model-o./output/example5And get the following output from in the log file:INFO:__main__:*** START listing files in 
INFO:__main__:
INFO:__main__:*** END file listing 
INFO:__main__:*** START listing files in /opt/ml/input/data/bucket
INFO:__main__:/opt/ml/input/data/bucket:
total 12
drwxr-xr-x 2 root root 4096 Sep 14 21:55 .
drwxr-xr-x 3 root root 4096 Sep 14 21:55 ..
-rw-r--r-- 1 root root  128 Sep 14 21:55 model.tar.gz

INFO:__main__:*** END file listing /opt/ml/input/data/bucketConfiguring the docker imageThe image used to run a task can either be selected from apre-built onesor extended with additional Dockerfile commands.
Theframework,framework_versionandpy_versionCLI parameters are used to define the pre-built image, then if a path to a directory containing the Dockerfile is given bydocker_file_path_or_content, it used along withaws_repo_name,repo_nameandimage_tagto build and push an image to ECS, and then set it as the used image.
The base image should be set to__BASE_IMAGE__within the Dockerfile, and is automatically replaced with the correct base image (according to the provided parameters above) before building it.
The API parameter for the Dockerfile path is nameddocker_file_path_or_contentand allows to provide the content of the Dockerfile, e.g.dockerFileContent="""# __BASE_IMAGE__ is automatically replaced with the correct base imageFROM __BASE_IMAGE__RUN pip3 install pandas==1.1 scikit-learn==0.21.3"""Sample usages:CLI Example 6_1- a CLI example launched byrun.shsingle file example- API with Dockerfile contentsingle task example- API with Dockerfile pathDefining code dependenciesAdditional local code dependencies can be specified with thedependenciesCLI/API parameters. These dependencies are packed along with
the source code, and are extracted on the root code folder in run time.Sample usages:CLI Example 6_2- a CLI example launched byrun.shsingle task example- APISingle file exampleAsingle file examplecan be found in theexamples directory.
First, define therunner:dockerFileContent="""# __BASE_IMAGE__ is automatically replaced with the correct base imageFROM __BASE_IMAGE__RUN pip3 install pandas==1.1 scikit-learn==0.21.3"""file_path=Path(__file__).parentdefrunner(project_name="simple-sagemaker-sf",prefix="",postfix="",output_path=None):fromsimple_sagemaker.sm_projectimportSageMakerProjectsm_project=SageMakerProject(project_name,prefix=prefix)# define the code parameterssm_project.setDefaultCodeParams(source_dir=None,entry_point=__file__,dependencies=[])# define the instance parameterssm_project.setDefaultInstanceParams(instance_count=2,max_run_mins=15)# docker imagesm_project.setDefaultImageParams(aws_repo_name="task_repo",repo_name="task_repo",image_tag="latest",docker_file_path_or_content=dockerFileContent,)image_uri=sm_project.buildOrGetImage(instance_type=sm_project.defaultInstanceParams.instance_type)# *** Task 1 - process input datatask1_name="task1"+postfix# set the input datainput_data_path=file_path/"data"# run the tasksm_project.runTask(task1_name,image_uri,input_distribution="ShardedByS3Key",# distribute the input files among the workershyperparameters={"worker":1,"arg":"hello world!","task":1},input_data_path=str(input_data_path)ifinput_data_path.is_dir()elseNone,clean_state=True,# clean the current state, also forces re-running)# download the resultsifnotoutput_path:output_path=file_path/"output"shutil.rmtree(output_path,ignore_errors=True)sm_project.downloadResults(task1_name,Path(output_path)/"output1")An additionaltaskthat depends on the previous one can now be scheduled as well:# *** Task 2 - process the results of Task 1task2_name="task2"+postfix# set the inputadditional_inputs={"task2_data":sm_project.getInputConfig(task1_name,"model"),"task2_data_dist":sm_project.getInputConfig(task1_name,"model",distribution="ShardedByS3Key"),}# run the tasksm_project.runTask(task2_name,image_uri,hyperparameters={"worker":1,"arg":"hello world!","task":2},clean_state=True,# clean the current state, also forces re-runningadditional_inputs=additional_inputs,)# download the resultssm_project.downloadResults(task2_name,Path(output_path)/"output2")returnsm_projectThen, the worker code (note: the same function is used for the two differenttasks, depending on thetaskhyperparameter):defworker():fromworker_toolkitimportworker_liblogger.info("Starting worker...")# parse the argumentsworker_config=worker_lib.WorkerConfig()logger.info(f"Hyperparams:{worker_config.hps}")logger.info(f"Input data files:{list(Path(worker_config.channel_data).rglob('*'))}")logger.info(f"State files:{list(Path(worker_config.state).rglob('*'))}")ifint(worker_config.hps["task"])==1:# update the state per running instanceopen(f"{worker_config.instance_state}/state_{worker_config.current_host}","wt").write("state")# write to the model output directoryforfileinPath(worker_config.channel_data).rglob("*"):iffile.is_file():relp=file.relative_to(worker_config.channel_data)path=Path(worker_config.model_dir)/(str(relp)+"_proc_by_"+worker_config.current_host)path.write_text(file.read_text()+" processed by "+worker_config.current_host)open(f"{worker_config.model_dir}/output_{worker_config.current_host}","wt").write("output")elifint(worker_config.hps["task"])==2:logger.info(f"Input task2_data:{list(Path(worker_config.channel_task2_data).rglob('*'))}")logger.info(f"Input task2_data_dist:{list(Path(worker_config.channel_task2_data_dist).rglob('*'))}")logger.info("finished!")# The task is marked as completedTo pack everything in a single file, we use the command line argument--worker(as defined in therunnerfunction) to distinguish betweenrunnerand worker runsimportloggingimportshutilimportsysfrompathlibimportPathlogger=logging.getLogger(__name__)...defmain():logging.basicConfig(stream=sys.stdout,level=logging.INFO)if"--worker"insys.argv:worker()else:runner()if__name__=="__main__":main()Running the file, with a sibling directory nameddatawith a sample fileas on the example, prduces the following outputs for Task 1:INFO:__main__:Hyperparams: {'arg': 'hello world!', 'task': 1, 'worker': 1}
INFO:__main__:Input data files: [PosixPath('/opt/ml/input/data/data/sample_data1.txt')]
INFO:__main__:State files: [PosixPath('/state/algo-1')]
INFO:worker_toolkit.worker_lib:Creating instance specific state dir
INFO:__main__:finished!INFO:__main__:Hyperparams: {'arg': 'hello world!', 'task': 1, 'worker': 1}
INFO:__main__:Input data files: [PosixPath('/opt/ml/input/data/data/sample_data2.txt')]
INFO:__main__:State files: [PosixPath('/state/algo-2')]
INFO:worker_toolkit.worker_lib:Creating instance specific state dir
INFO:__main__:finished!And the following for Task 2:INFO:__main__:Hyperparams: {'arg': 'hello world!', 'task': 2, 'worker': 1}
INFO:__main__:Input data files: [PosixPath('worker_toolkit'), PosixPath('example.py'), PosixPath('worker_toolkit/worker_lib.py'), PosixPath('worker_toolkit/__pycache__'), PosixPath('worker_toolkit/__init__.py'), PosixPath('worker_toolkit/__pycache__/__init__.cpython-38.pyc'), PosixPath('worker_toolkit/__pycache__/worker_lib.cpython-38.pyc')]
INFO:__main__:State files: [PosixPath('/state/algo-1')]
INFO:__main__:Input task2_data: [PosixPath('/opt/ml/input/data/task2_data/model.tar.gz')]
INFO:__main__:Input task2_data_dist: [PosixPath('/opt/ml/input/data/task2_data_dist/model.tar.gz')]
INFO:worker_toolkit.worker_lib:Creating instance specific state dirINFO:__main__:Hyperparams: {'arg': 'hello world!', 'task': 1, 'worker': 1}
INFO:__main__:Input data files: [PosixPath('/opt/ml/input/data/data/sample_data2.txt')]
INFO:__main__:State files: [PosixPath('/state/algo-2')]
INFO:worker_toolkit.worker_lib:Creating instance specific state dir
INFO:__main__:finished!As mentioned, the complete code can be found inthis directory,DevelopmentPushing a code changeDevelop ...Format & linttox-ecf
tox-elintCleanuptox-ecleanTesttoxGenerate & test coveragetox-ereport[Optionally] - bump the version string on /src/simple_sagemaker/initto allow the release of a new versionPush your code to a development branchEvery push is tested for linting + someCreate a pull request to the master branchEvery master push is fully testedIf the tests succeed, the new version is published toPyPiOpen issuesS3_sync doesn't delete remote files if deleted locally. Optimization may be needed as well.Bug: If arguments after "--" (followed by a space) are used, please initializeWorkerConfigobject withupdate_argv=True(the default) before parsing the command line arguments, e.g before callingparser.parse_args().Known issues: State directory can't have too many files (how many? seems to be a SageMaker bug. TBD: open a bug)Handling spot instance / timeout termination / signals (seems to be opened buga bug in deep-learning-containers)Full documentation of the APIs (Readme / Read the docs + CLI?)Add support for additional SageMaker features:Processing job (partial implementation exists)Built in algorithmsMoreframeworksExperimentsDebuggerAutomatic TuningJoin an in progress task if there's a need to run it / depend on its output