Package: sagemaker-containers
WARNING: This package has been deprecated. Please use theSageMaker Training Toolkitfor model training and theSageMaker Inference Toolkitfor model serving.SageMaker ContainersSageMaker Containers gives you tools to create SageMaker-compatible Docker containers, and has additional tools for letting you create Frameworks
(SageMaker-compatible Docker containers that can run arbitrary Python or shell scripts).Currently, this library is used by theSageMaker Scikit-learn containers.ContentsWARNING: This package has been deprecated. Please use the SageMaker Training Toolkit for model training and the SageMaker Inference Toolkit for model serving.SageMaker ContainersGetting StartedCreating a container using SageMaker ContainersThe DockerfileBuilding the containerTraining with Local ModeHow a script is executed inside the containerMapping hyperparameters to script argumentsReading additional information from the containerIMPORTANT ENVIRONMENT VARIABLESSM_MODEL_DIRSM_CHANNELSSM_CHANNEL_{channel_name}SM_HPSSM_HP_{hyperparameter_name}SM_CURRENT_HOSTSM_HOSTSSM_NUM_GPUSList of provided environment variables by SageMaker ContainersSM_NUM_CPUSSM_LOG_LEVELSM_NETWORK_INTERFACE_NAMESM_USER_ARGSSM_INPUT_DIRSM_INPUT_CONFIG_DIRSM_OUTPUT_DATA_DIRSM_RESOURCE_CONFIGSM_INPUT_DATA_CONFIGSM_TRAINING_ENVGetting StartedCreating a container using SageMaker ContainersHere we’ll demonstrate how to create a Docker image using SageMaker Containers in order to show the simplicity of using this library.Let’s suppose we need to train a model with the following training scripttrain.pyusing TF 2.0 in SageMaker:importtensorflowastfmnist=tf.keras.datasets.mnist(x_train,y_train),(x_test,y_test)=mnist.load_data()x_train,x_test=x_train/255.0,x_test/255.0model=tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),tf.keras.layers.Dense(128,activation='relu'),tf.keras.layers.Dropout(0.2),tf.keras.layers.Dense(10,activation='softmax')])model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])model.fit(x_train,y_train,epochs=1)model.evaluate(x_test,y_test)The DockerfileWe then create a Dockerfile with our dependencies and define the
program that will be executed in SageMaker:FROMtensorflow/tensorflow:2.0.0a0RUNpipinstallsagemaker-containers# Copies the training code inside the containerCOPYtrain.py/opt/ml/code/train.py# Defines train.py as script entry pointENVSAGEMAKER_PROGRAMtrain.pyMore documentation on how to build a Docker container can be foundhereBuilding the containerWe then build the Docker image usingdocker build:dockerbuild-ttf-2.0.Training with Local ModeWe can useLocal
Modeto test the container locally:fromsagemaker.estimatorimportEstimatorestimator=Estimator(image_name='tf-2.0',role='SageMakerRole',train_instance_count=1,train_instance_type='local')estimator.fit()After using Local Mode, we can push the image to ECR and run a SageMaker training job. To see a complete example on how to create a container using SageMaker
Container, including pushing it to ECR, see the example notebooktensorflow_bring_your_own.ipynb.How a script is executed inside the containerThe training script must be located under the folder/opt/ml/codeand its relative path is defined in the environment variableSAGEMAKER_PROGRAM. The following scripts are supported:Python scripts: uses the Python interpreter for any script with
.py suffixShell scripts: uses the Shell interpreter to execute any other
scriptWhen training starts, the interpreter executes the entry point, from the
example above:pythontrain.pyMapping hyperparameters to script argumentsAny hyperparameters provided by the training job will be passed by the
interpreter to the entry point as script arguments. For example the
training job hyperparameters:{"HyperParameters":{"batch-size":256,"learning-rate":0.0001,"communicator":"pure_nccl"}}Will be executed as:./user_script.sh--batch-size256--learning_rate0.0001--communicatorpure_ncclThe entry point is responsible for parsing these script arguments. For
example, in a Python script:importargparseif__name__=='__main__':parser=argparse.ArgumentParser()parser.add_argument('--learning-rate',type=int,default=1)parser.add_argument('--batch-size',type=int,default=64)parser.add_argument('--communicator',type=str)parser.add_argument('--frequency',type=int,default=20)args=parser.parse_args()...Reading additional information from the containerVery often, an entry point needs additional information from the
container that is not available inhyperparameters. SageMaker
Containers writes this information asenvironment variablesthat are
available inside the script. For example, the training job below
includes the channelstrainingandtesting:fromsagemaker.pytorchimportPyTorchestimator=PyTorch(entry_point='train.py',...)estimator.fit({'training':'s3://bucket/path/to/training/data','testing':'s3://bucket/path/to/testing/data'})The environment variableSM_CHANNEL_{channel_name}provides the
path were the channel is located:importargparseimportosif__name__=='__main__':parser=argparse.ArgumentParser()...# reads input channels training and testing from the environment variablesparser.add_argument('--training',type=str,default=os.environ['SM_CHANNEL_TRAINING'])parser.add_argument('--testing',type=str,default=os.environ['SM_CHANNEL_TESTING'])args=parser.parse_args()...When training starts, SageMaker Containers will print all available
environment variables.IMPORTANT ENVIRONMENT VARIABLESThese environment variables are those that you’re likely to use when
writing a user script. A full list of environment variables is given
below.SM_MODEL_DIRSM_MODEL_DIR=/opt/ml/modelWhen the training job finishes, the container will bedeletedincluding its file system withexceptionof the/opt/ml/modeland/opt/ml/outputfolders. Use/opt/ml/modelto save the model
checkpoints. These checkpoints will be uploaded to the default S3
bucket. Usage example:importos# using it in argparseparser.add_argument('model_dir',type=str,default=os.environ['SM_MODEL_DIR'])# using it as variablemodel_dir=os.environ['SM_MODEL_DIR']# saving checkpoints to model dir in chainerserializers.save_npz(os.path.join(os.environ['SM_MODEL_DIR'],'model.npz'),model)For more information, see:How Amazon SageMaker Processes Training
Output.SM_CHANNELSSM_CHANNELS='["testing","training"]'Contains the list of input data channels in the container.When you run training, you can partition your training data into
different logical “channels”. Depending on your problem, some common
channel ideas are: “training”, “testing”, “evaluation” or “images” and
“labels”.SM_CHANNELSincludes the name of the available channels in the
container as a JSON encoded list. Usage example:importosimportjson# using it in argparseparser.add_argument('channel_names',default=json.loads(os.environ['SM_CHANNELS'])))# using it as variablechannel_names=json.loads(os.environ['SM_CHANNELS']))SM_CHANNEL_{channel_name}SM_CHANNEL_TRAINING='/opt/ml/input/data/training'SM_CHANNEL_TESTING='/opt/ml/input/data/testing'Contains the directory where the channel namedchannel_nameis
located in the container. Usage examples:importosimportjsonparser.add_argument('--train',type=str,default=os.environ['SM_CHANNEL_TRAINING'])parser.add_argument('--test',type=str,default=os.environ['SM_CHANNEL_TESTING'])args=parser.parse_args()train_file=np.load(os.path.join(args.train,'train.npz'))test_file=np.load(os.path.join(args.test,'test.npz'))SM_HPSSM_HPS='{"batch-size": "256", "learning-rate": "0.0001","communicator": "pure_nccl"}'Contains a JSON encoded dictionary with the user provided
hyperparameters. Example usage:importosimportjsonhyperparameters=json.loads(os.environ['SM_HPS']))# {"batch-size": 256, "learning-rate": 0.0001, "communicator": "pure_nccl"}SM_HP_{hyperparameter_name}SM_HP_LEARNING-RATE=0.0001SM_HP_BATCH-SIZE=10000SM_HP_COMMUNICATOR=pure_ncclContains value of the hyperparameter namedhyperparameter_name.
Usage examples:learning_rate=float(os.environ['SM_HP_LEARNING-RATE'])batch_size=int(os.environ['SM_HP_BATCH-SIZE'])comminicator=os.environ['SM_HP_COMMUNICATOR']SM_CURRENT_HOSTSM_CURRENT_HOST=algo-1The name of the current container on the container network. Usage
example:importos# using it in argparseparser.add_argument('current_host',type=str,default=os.environ['SM_CURRENT_HOST'])# using it as variablecurrent_host=os.environ['SM_CURRENT_HOST']SM_HOSTSSM_HOSTS='["algo-1","algo-2"]'JSON encoded list containing all the hosts . Usage example:importosimportjson# using it in argparseparser.add_argument('hosts',type=str,default=json.loads(os.environ['SM_HOSTS']))# using it as variablehosts=json.loads(os.environ['SM_HOSTS'])SM_NUM_GPUSSM_NUM_GPUS=1The number of gpus available in the current container. Usage example:importos# using it in argparseparser.add_argument('num_gpus',type=int,default=os.environ['SM_NUM_GPUS'])# using it as variablenum_gpus=int(os.environ['SM_NUM_GPUS'])List of provided environment variables by SageMaker ContainersSM_NUM_CPUSSM_NUM_CPUS=32The number of cpus available in the current container. Usage example:# using it in argparseparser.add_argument('num_cpus',type=int,default=os.environ['SM_NUM_CPUS'])# using it as variablenum_cpus=int(os.environ['SM_NUM_CPUS'])SM_LOG_LEVELSM_LOG_LEVEL=20The current log level in the container. Usage example:importosimportlogginglogger=logging.getLogger(__name__)logger.setLevel(int(os.environ.get('SM_LOG_LEVEL',logging.INFO)))SM_NETWORK_INTERFACE_NAMESM_NETWORK_INTERFACE_NAME=ethweName of the network interface, useful for distributed training. Usage
example:# using it in argparseparser.add_argument('network_interface',type=str,default=os.environ['SM_NETWORK_INTERFACE_NAME'])# using it as variablenetwork_interface=os.environ['SM_NETWORK_INTERFACE_NAME']SM_USER_ARGSSM_USER_ARGS='["--batch-size","256","--learning_rate","0.0001","--communicator","pure_nccl"]'JSON encoded list with the script arguments provided for training.SM_INPUT_DIRSM_INPUT_DIR=/opt/ml/input/The path of the input directory, e.g./opt/ml/input/The input_dir,
e.g./opt/ml/input/, is the directory where SageMaker saves input
data and configuration files before and during training.SM_INPUT_CONFIG_DIRSM_INPUT_CONFIG_DIR=/opt/ml/input/configThe path of the input configuration directory, e.g./opt/ml/input/config/. The
directory where standard SageMaker configuration files are located, e.g./opt/ml/input/config/.SageMaker training creates the following files in this folder when
training starts:hyperparameters.json: Amazon SageMaker makes the hyperparameters in a CreateTrainingJob request available in this file.inputdataconfig.json: You specify data channel information in the InputDataConfig parameter in a CreateTrainingJob request. Amazon SageMaker makes this information available in this file.resourceconfig.json: name of the current host and all host containers in the training.More information about this files can be find here:https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.htmlSM_OUTPUT_DATA_DIRSM_OUTPUT_DATA_DIR=/opt/ml/output/data/algo-1The dir to write non-model training artifacts (e.g. evaluation results)
which will be retained by SageMaker, e.g./opt/ml/output/data.As your algorithm runs in a container, it generates output including the
status of the training job and model and output artifacts. Your
algorithm should write this information to the this directory.SM_RESOURCE_CONFIGSM_RESOURCE_CONFIG='{"current_host":"algo-1","hosts":["algo-1","algo-2"]}'The contents from/opt/ml/input/config/resourceconfig.json. It has
the following keys:current_host: The name of the current container on the container
network. For example,'algo-1'.hosts: The list of names of all containers on the container network,
sorted lexicographically. For example,['algo-1','algo-2','algo-3']for a three-node cluster.For more information aboutresourceconfig.json:https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html#your-algorithms-training-algo-running-container-dist-trainingSM_INPUT_DATA_CONFIGSM_INPUT_DATA_CONFIG='{
    "testing": {
        "RecordWrapperType": "None",
        "S3DistributionType": "FullyReplicated",
        "TrainingInputMode": "File"
    },
    "training": {
        "RecordWrapperType": "None",
        "S3DistributionType": "FullyReplicated",
        "TrainingInputMode": "File"
    }
}'Input data configuration from/opt/ml/input/config/inputdataconfig.json.For more information aboutinpudataconfig.json:https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html#your-algorithms-training-algo-running-container-dist-trainingSM_TRAINING_ENVSM_TRAINING_ENV='
{
    "channel_input_dirs": {
        "test": "/opt/ml/input/data/testing",
        "train": "/opt/ml/input/data/training"
    },
    "current_host": "algo-1",
    "framework_module": "sagemaker_chainer_container.training:main",
    "hosts": [
        "algo-1",
        "algo-2"
    ],
    "hyperparameters": {
        "batch-size": 10000,
        "epochs": 1
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {
        "test": {
            "RecordWrapperType": "None",
            "S3DistributionType": "FullyReplicated",
            "TrainingInputMode": "File"
        },
        "train": {
            "RecordWrapperType": "None",
            "S3DistributionType": "FullyReplicated",
            "TrainingInputMode": "File"
        }
    },
    "input_dir": "/opt/ml/input",
    "job_name": "preprod-chainer-2018-05-31-06-27-15-511",
    "log_level": 20,
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-{aws-region}-{aws-id}/{training-job-name}/source/sourcedir.tar.gz",
    "module_name": "user_script",
    "network_interface_name": "ethwe",
    "num_cpus": 4,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data/algo-1",
    "output_dir": "/opt/ml/output",
    "resource_config": {
        "current_host": "algo-1",
        "hosts": [
            "algo-1",
            "algo-2"
        ]
    }
}'Provides the entire training information as a JSON-encoded dictionary.