Package: cloudsend
DescriptionCloudSend is a Python package that allows you to run any script on a cloud service (for now AWS only).FeaturesEasily run scripts on AWS by writing a simple configuration fileHandles Python and Julia scripts, or any commandHandles PyPi , Conda/Mamba, Apt-get and Julia environmentsConcurrent instance supportHandles disconnections from instances, including stopped or terminated instancesHandles interruption of CloudSend, with state recoveryRuns locally or on a remote instance, with 'watcher' functionalityImportant NoteCloudSend helps you easily create instances on AWS so that you can focus on your scripts. It is important to realize that it can andwill likely generate extra costs. If you want to minimize those costs, activate theecomode in the configuration or make sure you monitor the resources created by CloudSend. Those include:VPCsSubnetsSecurity GroupsInstancesDevice Mappings / DisksPolicies & RolesKeyPairsPre-requisitesIn order to use the python AWS client (Boto3), you need to have an existing AWS account and to setup your computer for AWS.with AWS CLIGo tothe AWS Signup pageand create an accountDownloadthe AWS CLIIn the AWS web console,create a user with administrator privilegeIn the AWS web console, under the AMI section, click on the new user and make sure you create an access key under the tab "Security Credentials". Make sure "Console Password" is Enabled as wellIn ther Terminal, use the AWS CLI to setup your configuration:aws configureSeehttps://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.htmlTo run inremotemode, you also need toadd the following credentials to your user(maybe):iam:PassRoleiam:CreateRoleec2:AssociateIamInstanceProfileec2:ReplaceIamInstanceProfileAssociationmanuallyGo tothe AWS Signup pageand create an accountIn the AWS web console,create a user with administrator privilegeIn the AWS web console, under the IAM section, click on the new user and make sure you create an access key under the tab "Security Credentials". Make sure "Console Password" is Enabled as wellAdd your new user credentials manually,in the credentials file'~/.aws/config' example ('C:\Users\USERNAME\.aws\config' on Windows)[default]
region = eu-west-3
output = json'~/.aws/credentials' example ('C:\Users\USERNAME\.aws\credentials' on Windows)[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEYSetting up a separate user with least permissions (manually)In the AWS web console, in the IAM service, create a group 'cloudsend-users' withAmazonEC2FullAccessandIAMFullAccesspermissionsIn the AWS web console, in the IAM service, create a user USERNAME attached to the 'cloudsend-users' group:Step 1Step 2... Step 5COPY the Access Key info !Add your new user profile manually,in the credentials file'~/.aws/config' example ('C:\Users\USERNAME\.aws\config' on Windows)[default]
region = eu-west-3
output = json

[profile cloudsend]
region = eu-west-3
output = json'~/.aws/credentials' example ('C:\Users\USERNAME\.aws\credentials' on Windows)[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

[cloudsend]
aws_access_key_id = YOU_PROFILE_ACCESS_KEY_ID
aws_secret_access_key = YOUR_PROFILE_SECRET_ACCESS_KEYadd the 'profile' : 'cloudsend_USERNAME' to the configurationconfig={################################################################################# GLOBALS################################################################################'project':'test',# this will be concatenated with the instance hashes (if not None)'profile':'cloudsend',...InstallationOn MacOS / Linuxwith pippython3-mvenv.venvsource./.venv/bin/activate
python3-mensurepip--default-pip
python3-mpipinstall-rrequirements.txtwith poetrycurl-sSLhttps://install.python-poetry.org|python3.8-
poetryinstallOn Windows (powershell)with pipC:\> python3 -m venv .venvC:\> .venv\\Scripts\\activate.batC:\> python -m pip install -r requirements.txtwith poetryC:\> (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -C:\> poetry installUsage / Test runs# copy the example filecpexample/config.example.pyconfig.py## EDIT THE FILE## to run with pippython3-mcloudsend.democonfig# to run with pip with reset (maestro and instances)python3-mcloudsend.democonfigreset# to run with poetrypoetryrundemoconfig# to run with poetry with reset (maestro and the instances)poetryrundemoconfigresetConfiguration exampleconfig={################################################################################# GLOBALS################################################################################'project':'test',# this will be concatenated with the instance hashes (if not None)'profile':None,# if you want to use a specific profile (user/region), specify its name here'dev':False,# When True, this will ensure the same instance and dev environement are being used (while working on building up the project)'debug':1,# debug level (0...3)'maestro':'local',# where the 'maestro' resides: local' | 'remote' (micro instance)'auto_stop':True,# will automatically stop the instances and the maestro, once the jobs are done'provider':'aws',# the provider name ('aws' | 'azure' | ...)'job_assign':None,# algorithm used for job assignation / task scheduling ('random' | 'multi_knapsack')'recover':True,# if True, CloudSend will always save the state and try to recover this state on the next execution'print_deploy':False,# if True, this will cause the deploy stage to print more (and lock)'mutualize_uploads':True,# adjusts the directory structure of the uploads ... (False = per job or True = global/mutualized)################################################################################# INSTANCES / HARDWARE################################################################################'instances':[{'region':None,# can be None or has to be valid. Overrides AWS user region configuration.'cloud_id':None,# can be None, or even wrong/non-existing - then the default one is used'img_id':'ami-077fd75cd229c811b',# OS image: has to be valid and available for the profile (user/region)'img_username':'ubuntu',# the SSH user for the image'type':'t2.micro',# proprietary size spec (has to be valid)'cpus':None,# number of CPU cores'gpu':None,# the proprietary type of the GPU'disk_size':None,# the disk size of this instance type (in GB)'disk_type':None,# the proprietary disk type of this instance type: 'standard', 'io1', 'io2', 'st1', etc'eco':True,# eco == True >> SPOT e.g.'eco_life':None,# lifecycle of the machine in ECO mode (datetime.timedelta object) (can be None with eco = True)'max_bid':None,# max bid ($/hour) (can be None with eco = True)'number':1,# multiplicity: the number of instance(s) to create'explode':True# multiplicity: can this instance type be distributed accross multiple instances, to split CPUs}],################################################################################# ENVIRONMENTS / SOFTWARE################################################################################'environments':[{'name':None,# name of the environment - should be unique if not 'None'. 'None' only when len(environments)==1# env_conda + env_pypi  : mamba is used to setup the env (pip dependencies included)# env_conda (only)      : mamba is used to setup the env# env_pypi  (only)      : venv + pip is used to setup the env'command':'example/install_julia.sh',# None, or a string: path to a bash file to execute when deploying'env_aptget':["openssh-client"],# None, an array of librarires/binaries for apt-get'env_conda':"example/environment.yml",# None, an array of libraries, a path to environment.yml  file, or a path to the root of a conda environment'env_conda_channels':None,# None, an array of channels. If None (or absent), defaults and conda-forge will be used'env_pypi':"example/requirements.txt",# None, an array of libraries, a path to requirements.txt file, or a path to the root of a venv environment'env_julia':["Wavelets"],# None, a string or an array of Julia packages to install (requires julia)}],################################################################################# JOBS / SCRIPTS################################################################################'jobs':[{'env_name':None,# the environment to use (can be 'None' if solely one environment is provided above)'cpus_req':None,# the CPU(s) requirements for the process (can be None)'run_script':'example/run_remote.py 1 10',# the script to run (Python (.py) or Julia (.jl) for now) (prioritised vs 'run_command')'run_command':None,# the command to run'upload_files':["uploaded.txt"],# any file to upload (array or string) - will be put in the same directory'input_files':'input.dat',# the input file name (used by the script)'output_files':'output.dat',# the output file name (used by the script)'repeat':2,# the number of times this job is repeated},{'env_name':None,# the environment to use (can be 'None' if solely one environment is provided above)'cpus_req':None,# the CPU(s) requirements for the process (can be None)'run_script':'example/run_remote.py 2 12',# the script to run (Python (.py) or Julia (.jl) for now) (prioritised vs 'run_command')'run_command':None,# the command to run'upload_files':["uploaded.txt"],# any file to upload (array or string) - will be put in the same directory'input_files':'input.dat',# the input file name (used by the script)'output_files':'output.dat',# the output file name (used by the script)}]}Minimum configuration exampleconfig={'debug':1,# debug level (0...3)'maestro':'local',# where the 'maestro' resides: local' | 'remote' (nano instance) | 'lambda''provider':'aws',# the provider name ('aws' | 'azure' | ...)'instances':[{'type':'t2.micro',# proprietary size spec (has to be valid)}],'environments':[{'name':None,# name of the environment - should be unique if not 'None'. 'None' only when len(environments)==1'env_conda':"example/environment.yml",# None, an array of libraries, a path to environment.yml  file, or a path to the root of a conda environment'env_julia':["Wavelets"],# None, a string or an array of Julia packages to install (requires julia)}],'jobs':[{'env_name':None,# the environment to use (can be 'None' if solely one environment is provided above)'cpus_req':None,# the CPU(s) requirements for the process (can be None)'run_script':'example/run_remote.py 1 10',# the script to run (Python (.py) or Julia (.jl) for now) (prioritised vs 'run_command')'upload_files':["uploaded.txt"],# any file to upload (array or string) - will be put in the same directory'input_files':'input.dat',# the input file name (used by the script)'output_files':'output.dat',# the output file name (used by the script)},{'env_name':None,# the environment to use (can be 'None' if solely one environment is provided above)'cpus_req':None,# the CPU(s) requirements for the process (can be None)'run_script':'example/run_remote.py 2 12',# the script to run (Python (.py) or Julia (.jl) for now) (prioritised vs 'run_command')'upload_files':["uploaded.txt"],# any file to upload (array or string) - will be put in the same directory'input_files':'input.dat',# the input file name (used by the script)'output_files':'output.dat',# the output file name (used by the script)}]}Python APIclassCloudSendLightProvider(ABC):classCloudSendFatProvider(ABC):defdebug(self,level,*args,**kwargs):# start the provider: creates the instances# if reset = True, CloudSend forces a process cleanup as well as more re-uploadsdefstart(self,reset):# deploy all materials (environments, files, scripts etc.)defdeploy(self):# run the jobs# returns a CloudSendRunSessiondefrun(self,wait=False):# wait for the processes to reach a statedefwait(self,job_state,run_session=None):# get the states of the processesdefget_jobs_states(self,run_session=None):# print a summary of processesdefprint_jobs_summary(self,run_session=None,instance=None):# print the aborted logs, if anydefprint_aborted_logs(self,run_session=None,instance=None):# fetch results datadeffetch_results(self,out_directory=None,run_session=None):# wait for the watcher process to be completely done (useful for demo)deffinalize(self):# wakeup = start + assign + deploy + run + watchdefwakeup(self)@abstractmethoddefget_region(self):@abstractmethoddefget_recommended_cpus(self,inst_cfg):@abstractmethoddefcreate_instance_objects(self,config,for_maestro):@abstractmethoddeffind_instance(self,config):@abstractmethoddefstart_instance(self,instance):@abstractmethoddefstop_instance(self,instance):@abstractmethoddefterminate_instance(self,instance):@abstractmethoddefupdate_instance_info(self,instance):# GLOBAL methodsdefget_client(provider='aws',maestro='local')CloudSend usagePython programmatic useNote: this demo works the same way, whether CloudSend runs locally or remotelyfromcloudsendimportproviderascloudsendfromcloudsend.coreimportCloudSendProcessStateimportasyncio# load configconfig=__import__(config).config# create provider: this loads the configprovider=cloudsend.get_client(config)# start the provider: this attempts to create the instancesawaitprovider.start()# deploy the necessary stuff onto the instancesawaitprovider.deploy()# run the jobs and get active processes objects backrun_session=awaitprovider.run()# wait for the active proccesses to be done or aborted:awaitprovider.wait(CloudSendProcessState.DONE|CloudSendProcessState.ABORTED)# you can get the state of all jobs this way:awaitprovider.get_jobs_states()# or get the state for a specific run session:awaitprovider.get_jobs_states(run_session)# you can print processes summary with:awaitprovider.print_jobs_summary()# get the results file locallyawaitprovider.fetch_results('./tmp')CLI useNote: the commands below work the same way, whether CloudSend runs locally or remotelywith Poetry# init the client with global params and add instances, envs and jobs (if any)poetryruncliinitconfig.py# add more jobspoetryrunclicfg_add_jobsconfig_jobs.py# add more stuffpoetryrunclicfg_add_configconfig_more.py# deploy the material onto the instancespoetryrunclideploy# run the jobspoetryrunclirun# wait for the jobs to be donepoetryruncliwait# get the resultspoetryrunclifetch_results# shutdown the daemonpoetryrunclishutdownwith virtualenv# init the client with global params and add instances, envs and jobs (if any)python3-mcloudsend.cliinitconfig.py# add more jobspython3-mcloudsend.clicfg_add_jobsconfig_jobs.py# add more stuffpython3-mcloudsend.clicfg_add_configconfig_more.py# deploy the material onto the instancespython3-mcloudsend.clideploy# run the jobspython3-mcloudsend.clirun# wait for the jobs to be donepython3-mcloudsend.cliwait# get the resultspython3-mcloudsend.clifetch_results# shutdown the daemonpython3-mcloudsend.clishutdownContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT