Package: valohai-utils
valohai-utilsPython helper library for the Valohai machine learning platform.Installpipinstallvalohai-utilsExecutionRun locallypythonmycode.pyRun in the cloudvhyamlstepmycode.py
vhexecrun-amystepWhat does valohai-utils do?Generates and updates thevalohai.yamlconfiguration file based on the source codeAgnostic input handling (single file, multiple files, zip, tar)Parse command-line parametersCompress outputsDownload inputs for local experimentsStraightforward way to print metrics as Valohai metadataCode parity between local vs. cloudParametersValohai parametersare variables & hyper-parameters that are parsed from the command-line. You define parameters in a dictionary:default_parameters={'iterations':100,'learning_rate':0.001}The dictionary is fed tovalohai.prepare()method:The values given aredefaultvalues. You can override them from the command-line or using the Valohai web UI.Exampleimportvalohaidefault_parameters={'iterations':10,}valohai.prepare(step="helloworld",default_parameters=default_parameters)foriinrange(valohai.parameters('iterations').value):print("Iteration%s"%i)InputsValohai inputsare the data files required by the experiment. They are automatically downloaded for you, if the data is from a public source. You define inputs with a dictionary:default_inputs={'input_name':'http://example.com/1.png'}An input can also be a list of URLs or a folder:default_inputs={'input_name':['http://example.com/1.png','http://example.com/2.png'],'input_folder':['s3://mybucket/images/*','azure://mycontainer/images/*','gs://mybucket/images/*']}Or it can be an archive full of files (uncompressed automagically on-demand):default_inputs={'images':'http://example.com/myimages.zip'}The dictionary is fed tovalohai.prepare()method.The url(s) given aredefaults. You can override them from the command-line or using the Valohai web UI.Exampleimportcsvimportvalohaidefault_inputs={'myinput':'https://pokemon-images-example.s3-eu-west-1.amazonaws.com/pokemon.csv'}valohai.prepare(step="test",default_inputs=default_inputs)withopen(valohai.inputs("myinput").path())ascsv_file:reader=csv.reader(csv_file,delimiter=',')OutputsValohai outputsare the files that your step produces an end result.When you are ready to save your output file, you can query for the correct path from thevalohai-utils.Exampleimage=Image.open(in_path)new_image=image.resize((width,height))out_path=valohai.outputs('resized').path('resized_image.png')new_image.save(out_path)Sometimes there are so many outputs that you may want to compress them into a single file.In this case, once you have all your outputs saved, you can finalize the output with thecompress()method.Examplevalohai.outputs('resized').compress("*.png","images.zip",remove_originals=True)LoggingYou can log metrics using theValohai metadata systemand then render interactive graphs on the web interface. Thevalohai-utilslogger will print JSON logs that Valohai will parse as metadata.It is important for visualization that logs for single epoch are flushed out as a single JSON object.Exampleimportvalohaiforepochinrange(100):withvalohai.metadata.logger()aslogger:logger.log("epoch",epoch)logger.log("accuracy",accuracy)logger.log("loss",loss)Example 2importvalohailogger=valohai.logger()forepochinrange(100):logger.log("epoch",epoch)logger.log("accuracy",accuracy)logger.log("loss",loss)logger.flush()Distributed Workloadsvalohai.distributedcontains a toolset for running distributed tasks on Valohai.importvalohaiifvalohai.distributed.is_distributed_task():# `master()` reports the same worker on all contextsmaster=valohai.distributed.master()master_url=f'tcp://{master.primary_local_ip}:1234'# `members()` contains all workers in the distributed taskmember_public_ips=",".join([m.primary_public_ipforminvalohai.distributed.members()])# `me()` has full details about the current worker contextdetails=valohai.distributed.me()size=valohai.distributed.required_countrank=valohai.distributed.rank# 0, 1, 2, etc. depending on run contextFull examplePreprocess step for resizing image filesThis example step will do the following:Take image files (or an archive containing images) as input.Resize each image to the size provided by the width & height parameters.Compress the resized images intoresized/images.zipValohai output file.importosimportvalohaifromPILimportImagedefault_parameters={"width":640,"height":480,}default_inputs={"images":["https://dist.valohai.com/valohai-utils-tests/Example.jpg","https://dist.valohai.com/valohai-utils-tests/planeshark.jpg",],}valohai.prepare(step="resize",default_parameters=default_parameters,default_inputs=default_inputs)defresize_image(in_path,out_path,width,height,logger):image=Image.open(in_path)logger.log("from_width",image.size[0])logger.log("from_height",image.size[1])logger.log("to_width",width)logger.log("to_height",height)new_image=image.resize((width,height))new_image.save(out_path)if__name__=='__main__':forimage_pathinvalohai.inputs('images').paths():withvalohai.metadata.logger()aslogger:filename=os.path.basename(image_path)resize_image(in_path=image_path,out_path=valohai.outputs('resized').path(filename),width=valohai.parameters('width').value,height=valohai.parameters('height').value,logger=logger)valohai.outputs('resized').compress("*","images.zip",remove_originals=True)CLI command:vh yaml step resize.pyWill produce thisvalohai.yamlconfig:-step:name:resizeimage:python:3.7command:python ./resize.py {parameters}parameters:-name:widthdefault:640multiple-separator:','optional:falsetype:integer-name:heightdefault:480multiple-separator:','optional:falsetype:integerinputs:-name:imagesdefault:-https://dist.valohai.com/valohai-utils-tests/Example.jpg-https://dist.valohai.com/valohai-utils-tests/planeshark.jpgoptional:falseConfigurationThere are some environment variables that affect howvalohai-utilsworks when not running within a Valohai execution context.VH_FLAT_LOCAL_OUTPUTSIf set, flattens the local outputs directory structure into a single directory.
This means that outputs from subsequent runs can clobber old files.DevelopmentIf you wish to further developvalohai-utils, remember to install development dependencies and write tests for your additions.LintingLints are run via pre-commit.If you want pre-commit to check your commits via git hooks,pip install pre-commit
pre-commit installYou can also run the lints manually withpre-commit run --all-files.Testingpipinstall-e.-rrequirements-dev.txt
pytest