Package: sagemaker-experiments
NOTE:Use the SageMakerSDKto use SageMaker Experiments. This repository will not be up to date with the latest product improvements. Link todeveloper guide.SageMaker Experiments Python SDKExperiment tracking in SageMaker Training Jobs, Processing Jobs, and Notebooks.OverviewSageMaker Experiments is an AWS service for tracking machine learning Experiments. The SageMaker Experiments Python SDK is a high-level interface to this service that helps you track Experiment information using Python.Experiment tracking powers the machine learning integrated development environmentAmazon SageMaker Studio.For detailed API reference please go to:Read the DocsConceptsExperiment: A collection of related Trials. Add Trials to an Experiment that you wish to compare together.Trial: A description of a multi-step machine learning workflow. Each step in the workflow is described by a Trial Component. There is no relationship between Trial Components such as ordering.Trial Component: A description of a single step in a machine learning workflow.  For example data cleaning, feature extraction, model training, model evaluation, etc…Tracker: A Python context-manager for logging information about a single TrialComponent.For more information seeAmazon SageMaker Experiments - Organize, Track, and Compare Your Machine Learning TrainingsUsing the SDKYou can use this SDK to:Manage Experiments, Trials, and Trial Components within Python scripts, programs, and notebooks.Add tracking information to a SageMaker notebook, allowing you to model your notebook in SageMaker Experiments as a multi-step ML workflow.Record experiment information from inside your running SageMaker Training and Processing Jobs.Installationpipinstallsagemaker-experimentsExamplesimportboto3importpickle,gzip,numpy,json,osimportioimportnumpyasnpimportsagemaker.amazon.commonassmacimportsagemakerfromsagemakerimportget_execution_rolefromsagemakerimportanalyticsfromsmexperimentsimportexperiment# Specify training containerfromsagemaker.amazon.amazon_estimatorimportget_image_uricontainer=get_image_uri(boto3.Session().region_name,'linear-learner')# Load the datasets3=boto3.client("s3")s3.download_file("sagemaker-sample-files","datasets/image/MNIST/mnist.pkl.gz","mnist.pkl.gz")withgzip.open('mnist.pkl.gz','rb')asf:train_set,valid_set,test_set=pickle.load(f,encoding='latin1')vectors=np.array([t.tolist()fortintrain_set[0]]).astype('float32')labels=np.where(np.array([t.tolist()fortintrain_set[1]])==0,1,0).astype('float32')buf=io.BytesIO()smac.write_numpy_to_dense_tensor(buf,vectors,labels)buf.seek(0)key='recordio-pb-data'bucket=sagemaker.session.Session().default_bucket()prefix='sagemaker/DEMO-linear-mnist'boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix,'train',key)).upload_fileobj(buf)s3_train_data='s3://{}/{}/train/{}'.format(bucket,prefix,key)output_location='s3://{}/{}/output'.format(bucket,prefix)my_experiment=experiment.Experiment.create(experiment_name='MNIST')my_trial=my_experiment.create_trial(trial_name='linear-learner')role=get_execution_role()sess=sagemaker.Session()linear=sagemaker.estimator.Estimator(container,role,train_instance_count=1,train_instance_type='ml.c4.xlarge',output_path=output_location,sagemaker_session=sess)linear.set_hyperparameters(feature_dim=784,predictor_type='binary_classifier',mini_batch_size=200)linear.fit(inputs={'train':s3_train_data},experiment_config={"ExperimentName":my_experiment.experiment_name,"TrialName":my_trial.trial_name,"TrialComponentDisplayName":"MNIST-linear-learner",},)trial_component_analytics=analytics.ExperimentAnalytics(experiment_name=my_experiment.experiment_name)analytic_table=trial_component_analytics.dataframe()analytic_tableFor more examples, check out:sagemaker-experimentsinAWS Labs Amazon SageMaker Examples.LicenseThis library is licensed under the Apache 2.0 License.Running TestsUnit Teststoxtests/unitIntegration TestsTo run the integration tests, the following prerequisites must be met:AWS account credentials are available in the environment for the boto3 client to use.The AWS account has an IAM role with SageMaker permissions.toxtests/integTest against different regionstox-epy39----regioncn-north-1Docker Based Integration TestsSeveral integration tests rely on docker to push an image to ECR which is then used for training.Docker SetupInstall dockerset aws cred helper in docker config (~/.docker/config.json)#dockerconfigexample{"stackOrchestrator":"swarm","credsStore":"desktop","auths":{"https://index.docker.io/v1/":{}},"credHelpers":{"aws_account_id.dkr.ecr.region.amazonaws.com":"ecr-login"},"experimental":"disabled"}# run only docker based teststox-epy39--tests/integ-m'docker'# exclude docker based teststox-epy39--tests/integ-m'not docker'Generate Docstox-edocs