Package: sagemaker-rightline
sagemaker-rightlineThis repository contains the source code for sagemaker-rightline, a Python package that eases validation of properties of a SageMaker Pipeline object.Note that at present this package is in an early stage of development and is not yet ready for production use. We welcome contributions!README ContentFeaturesUsageContributingFeatures⚙️ ConfigurationTheConfigurationclass is responsible for running theValidationsagainst thePipelineobject and returning aReport.
TheConfigurationclass is instantiated with asagemaker.workflow.pipeline.Pipelineobject, anda list ofValidations.✔️ ValidationsAValidationis a class that inherits from theValidationbase class.
It is responsible for validating a single property of thePipelineobject.
We differentiate betweenValidationsthat check thePipelineobject itself (class names beginning with "Pipeline") andValidationsthat check thePipelineobject'sStepobjects (class name starting with "Step").
Depending on the specificValidation, a different set ofStepTypEnumsmay be supported.For example, theStepImagesExistsupportsProcessingandTrainingsteps. It's a validation checks that all ImageURI that
Steps of the named types of thePipelineobject reference indeed exist on the target ECR.The followingValidationsare currently implemented:PipelineParametersAsExpectedStepImagesExistStepKmsKeyIdAsExpectedStepNetworkConfigAsExpectedStepLambdaFunctionExistsStepRoleNameExistsStepRoleNameAsExpectedStepTagsAsExpectedStepInputsAsExpectedStepOutputsAsExpectedStepOutputsMatchInputsAsExpectedStepCallbackSqsQueueExistsPipelineProcessingStepsIONamesUniqueIn most cases, aValidationsubclass requires passing aRuleobject to its constructor.📜 RulesARuleis a class that inherits from theRulebase class.
It is responsible for defining the rule that aValidationchecks for.
For example, passing the list of expected KMSKeyIDs and theRuleEqualstoStepKmsKeyIdAsExpectedwill check that
allStepobjects of thePipelineobject have aKmsKeyIdproperty that matches the passed KMSKeyIDs.Note that not allValidationsrequire aRuleobject, e.g.StepImagesExist.The followingRulesare currently implemented:EqualsContainsAll rules support thenegativeparameter (default:False), which allows for inverting the rule.📝 ReportAReportis a class whose instance is returned by theConfigurationclass (optionally a pandas.DataFrame instead).
It contains the results of theValidationsthat were run against thePipelineobject as well as additional information
to allow for further analysis.Usagefromsagemaker.processingimportNetworkConfig,ProcessingInput,ProcessingOutputfromsagemaker.workflow.parametersimportParameterStringfromsagemaker_rightline.modelimportConfigurationfromsagemaker_rightline.rulesimportContains,Equalsfromsagemaker_rightline.validationsimport(PipelineParametersAsExpected,StepImagesExist,StepKmsKeyIdAsExpected,StepNetworkConfigAsExpected,StepLambdaFunctionExists,StepRoleNameExists,StepRoleNameAsExpected,StepTagsAsExpected,StepInputsAsExpected,StepOutputsAsExpected,StepOutputsMatchInputsAsExpected,StepCallbackSqsQueueExists,)# Import a dummy pipelinefromtests.fixtures.pipelineimportget_sagemaker_pipeline,DUMMY_BUCKETsm_pipeline=get_sagemaker_pipeline()# Define Validationsvalidations=[StepImagesExist(),PipelineParametersAsExpected(parameters_expected=[ParameterString(name="parameter-1",default_value="some-value",),],rule=Contains(),),StepKmsKeyIdAsExpected(kms_key_id_expected="some/kms-key-alias",step_name="sm_training_step_sklearn",# optional: if not set, will check all stepsrule=Equals(),),StepNetworkConfigAsExpected(network_config_expected=NetworkConfig(enable_network_isolation=False,security_group_ids=["sg-1234567890"],subnets=["subnet-1234567890"],),rule=Equals(negative=True),),StepLambdaFunctionExists(),StepRoleNameExists(),StepRoleNameAsExpected(role_name_expected="some-role-name",step_name="sm_training_step_sklearn",# optional: if not set, will check all stepsrule=Equals(),),StepTagsAsExpected(tags_expected=[{"some-key":"some-value",}],step_name="sm_training_step_sklearn",# optional: if not set, will check all stepsrule=Equals(),),StepInputsAsExpected(inputs_expected=[ProcessingInput(source=f"s3://{DUMMY_BUCKET}/input-1",destination="/opt/ml/processing/input",input_name="input-2",)],step_type="Processing",# either step_type or step_name must be set to filterrule=Contains(),),StepOutputsAsExpected(outputs_expected=[ProcessingOutput(source="/opt/ml/processing/output",destination=f"s3://{DUMMY_BUCKET}/output-1",output_name="output-1",)],step_name="sm_processing_step_spark",# optionalrule=Contains(),),StepOutputsMatchInputsAsExpected(inputs_outputs_expected=[{"input":{"step_name":"sm_processing_step_sklearn","input_name":"input-1",},"output":{"step_name":"sm_processing_step_sklearn","output_name":"output-1",},}]),StepCallbackSqsQueueExists(),]# Add Validations and SageMaker Pipeline to Configurationcm=Configuration(validations=validations,sagemaker_pipeline=sm_pipeline,)# Run the full Configurationdf=cm.run()# Show the reportdfReleasePublishing a new version to PyPI is done via theReleasefunctionality.
This will trigger thepublish.ymlworkflow, creating a new release with the version from the tag and publish the package to PyPI.ContributingContributions welcome! We'll add a guide shortly.
To get an overview of the structure of the project, have a look at theclass diagram, which is auto-generated at build time together with the corresponding PUML file.