Package: amlctor
AMLCTORAzure Machine Learning Pipeline Constructoramlctorallows you to create Azure Machine Learning(shortly -AML)Pipeline.amlctorbased on theAzure Machine Learning SDK, and implements main operations of the Pipeline creation. You can create pipelines with AML Steps, which can take DataInputs.
In amlctor pipeline creation consists of 3 steps:0. PreporationIt's highly recommended to create separated folder your pipeline projects. And also, virtual environment(venv). You can create separated venv for future AML projects. It's specially useful if you are working with different kinds of libraries: data science oriented, web and so on.1. Pipeline initialisationSomething like project initialisation. You choose pipeline name, directory and credential.envfile. For storing amlctor has denv storage - orEnvBank. Initialise pipeline as:python-mamlctorinit-nmyfirstpipe-p.-edenv_nameHere-nshows pipeline name,-p- directory in which pipeline will be created,-e- dotenv name. I will talk about denv's a little bit later. After this, in the passed directory will be created named as pipeline passed name.myfirstpipe
---|settings/
------|settings.py
------|.amlignore
------|.env
------|conda_dependencies.ymlInside the directorysettingsdirectory which contains:settings.py,.amlignore,.envandconda_dependencies.ymlfiles.conda_dependencies.ymlwill be used for environment creation on AML side..amlignoresomething like.gitignorebut for AML..envis file form of our EnvBank instance.-eis optional, if it's skipped, will be created.envtemplate with necessary fields, which you have to fill beforerunningpipeline.settings.py:This module contains all necessary configuractions:fromamlctor.inputimportFileInputSchema,PathInputSchemafromamlctor.coreimportStepSchema# --------------------------| Module Names |----------------------------AML_MODULE_NAME:str='aml'SCRIPT_MODULE_NAME:str='script'DATALOADER_MODULE_NAME:str='data_loader'# ---------------------------| General |---------------------------------NAME="{{pipe_name}}"DESCRIPTION="Your pipeline description"# ---------------------------| DataInputs |-------------------------------file=FileInputSchema(name='name',datastore_name='datastore',path_on_datastore='',files=['file.ext'],data_reference_name='')path=PathInputSchema(name='name',datastore_name='datastore',path_on_datastore='',data_reference_name='')# ---------------------------| Steps |---------------------------------step1=StepSchema(name='step_name',compute_target='compute_name',input_data=[file,path],allow_reuse=False)STEPS=[step1,]# ---------------------------| extra |---------------------------------EXTRA={'continue_on_step_failure':False,}Lets look at the variables we have here.AML_MODULE_NAME- initially, pipeline project has 3 main scripts:dataloader.py- loads all the DataInputs into the pipeline,aml.py- main script of the pipeline, loaded data inputs imported here automaticaly,script.py- just empty script for implement your deep logic. You are free for remove this module or add so many as you need, however - the entry point of project isaml.py.AML_MODULE_NAMEis the name of aml.py module. And the same thing forDATALOADER_MODULE_NAMEandSCRIPT_MODULE_NAME.NAME- name of your pipeline.DESCRIPTION- description of the pipeline.PathInputSchemaandFileInputSchemaDataInput of your pipeline. You create instances of the classes and pass intoStepSchemaclass. EachStepSchemaclass is abstraction ofPythonScriptStep. All steps must be insideSTEPSlist.After filling settings, you have to apply your settings.2.ApplySettingspython-mamlctorapply-p<path_to_pipeline>Applying pipeline means - create structure based on thesettings.pymodule. For each step will be created directory inside pipeline directory and each directory will contain:aml.py,dataloader.pyandscript.py.Note: names of the modules setted in thesettings.pymodule.3.RunPipelinepython-mamlctorrun-p<path_to_pipeline>This command will publish your pipeline into your AML.EnvBankFor work on AML pipeline you have to use your credentials:workspace_name,resource_group,subscription_id,build_id,environment_nameandtenant_id. In amltor these variables store as instances ofEnvBank, which is encrypted jsonlike file. You can create, retrieve or removeEnvBankinstances(I'll name it asdenv). In this purpose you've to usedenvcommand.Create denvYou can create denv in 2 ways: pass path of existing.envfile or in interactive mode - via terminal. In the first case:python-mamlctordenvcreate-p<path_to_.envfile>-n<new_name>Then you'll type new password twise for encryption. After that, denv will save into local storage and you will be able to use it for future pipeline creation.For create denv in interactive mode, you have to pass-ior--interactivearg:python-mamlctordenvcreate-iAfter that you have to type each asked field and set password.Get denvFor retrieve denv use:python-mamlctordenvget-n<name_of_denv>For list all existing denv names add --allargument:python-mamlctordenvget--allNote:for view the denv, you have to type password.Remove denvFor removing denv:python-mamlctordenvrm-n<name_of_denv>DataInputsDataInputs can be files or paths fromAML Datastore. Whole process is creatingDataReferenceobject behind the scenes... All inputs will be loaded in thedataloader.pyand imported intoaml.pymodule. Lets look atamlctorDataInputs.PathInputSchemaAllows you to create data reference link to any directory inside the datastore. class looks like this:classPathInputSchema:name:strdatastore_name:strpath_on_datastore:strdata_reference_name:strWhere:namename of your PathInput, this name will be used as variable name for importing.datastore_name- Datastore name,path_on_datastore- target path related to the Datastore.data_reference_name- data reference name forDataReferenceclass, optional - if empty, will be used name.FileInputSchemaAllows you to mount files from Datastore. Behind the scines, very similar to PathInput, but withfile orientedadditions.classFileInputSchema:name:strdatastore_name:strpath_on_datastore:strdata_reference_name:strfiles:List[str]First 4 fields as previous.files- you can list file or files as list, which will be mounted from Datastore. If you want to get one file, pass as string, for more files - list of strings. When you pass multiple filename, they must be on the same path.Supported file types:amlctorusespandasread methods for read the mounted files. At the moment, suported file types:csv, parquet, excell sheet, jsonSlugged file names will be used as variable names for importing files.Other commandsUpdateYou can updatedataloaderaccording to thesettings.pymodule. It can be useful when you maked some changes intosettings.pyand don't want to overwrite whole pipeline structure by scratch, in this case you can useupdate:python-mamlctorupdate-p<path_to_pipe>-sstep_name[Optional]step_nameargument is optional, if not passed, updating will apply for all steps, otherwise - only for passed step.Renamepython-mamlctorrename-p<path_to_pipe>-n<new_name>Renames pipeline intonew_name. Renaming pipeline means: rename pipeline project directory, changeNAMEvariable insettings.pyand editENVIRONMENT_FILEin the.envfile.