Package: biobb-remote
biobb_remoteIntroductionBiobb_remote is a package to allow biobb's to be executed on remote sites
through sshssh_credentials.pyProvides SSHCredentials to manage the generation, and installation of ssh credentialscredentials = SSHCredentials(host='', userid='', generate_key=False, look_for_keys=True)host (str): remote host nameuserid (str): remote user namegenerate_key (bool): Generates new public/private keys pairlook_for_keys (bool): Look user's .ssh keys if keys not setMethods(void) credentials.save(output_path, public_key_path=None, private_key_path=None, passwd=None)Stores SSHCredentials object in a external fileoutput_path (str): Path to filepublic_key_path (str): Path to a standard public key fileprivate_key_path (str): Path to a standard private key filepasswd (str): Password to encrypt private key (optional)(void) credentials.load_from_file(credentials_path, passwd=None)Recovers SSHCredentials object from disk filecredentials_path (str): Path to packed credentials file.passwd (str): Passwd to decrypt private key (optional)(void) credentials.load_from_private_key_file(private_path, passwd=None)Recovers SSHCredentials object from disk fileprivate_path (str): Path to private key file.passwd (str): Passwd to decrypt private key (optional)(void) credentials.generate_key(nbits=2048)Generates RSA keys pairnbits (int): number of bits the generated key(str) credentials.get_public_key(suffix='@biobb')Returns a readable publik key suitable to addto authorized keyssuffix (str): Added to the key for identify it.(str) credentials.get_private_key(passwd=None)Returns a readable possibly encrypted private keypasswd (str): Password to encrypt private key (optional)(bool) credentials.check_host_auth()Checks for public_key in remote .ssh/authorized_keys file. Requires users' SSH access to host(void) credentials.install_host_auth(file_bck='bck')Installs public_key on remote .ssh/authorized_keys file. Requires users' SSH access to hostfile_bck (str): Generates an authorized.file_bckfile with original authorized keys(void) credentials.remove_host_auth(file_bck='biobb')Removes public_key on remote .ssh/authorized_keys file. Requires users' SSH access to hostfile_bck (str): Generates an authorized.file_bckfile with original authorized keysssh_session.pyClass wrapping ssh operationsssh_session = SSHSession(ssh_data=None, credentials_path=None, private_path=None, passwd=None)ssh_data (SSHCredentials) : SSHCredentials objectcredentials_path (str) : Path to packed credentials file to useprivate_path (str): Path to private key filepasswd (str): Password to decrypt credentials (optional)(str) ssh_session.run_command(command)Runs command on remote. Returns stdout + stderrcommand (str): Command line to execute(bool | file_handle) ssh_session.run_sftp(oper, input_file_path, output_file_path='')Runs SFTP session on remoteoper (str): Operation to perform, one of
* get (gets a single file from input_file_path (remote) to output_file_path (local) )
* put (puts a single file from input_file_path (local) to output_file_path (remote)
* create (creates a file in output_file_path (remote) from input_file_path string-
* file (opens a remote file in input_file_path for read). Returns a file handle.
* listdir (returns a list of files in remote input_file_pathinput_file_path (str): Input file path or input stringoutput_file_path (str): Output file pathtask.pyDataBundleClass to manage bundles of input/output filesdata_bundle = DataBundle(bundle_id)bundle_id (str): Id for the data bundledata_bundle.add_file(file_path)Adds a single file to the data bundlefile_path (str): Path to the file to adddata_bundle.add_dir(dir_path)Adds all files from a directorydir_path (str): Path to the directory to add([str]) data_bundle.get_file_names()Generates a list of names or included files(str) data_bundle.to_json()Generates a Json dumpTaskAbstract module to handle remote tasks. Not for direct use, extend to include specific queueing systemsConstantstask.UNKNOWN = 0
task.SUBMITTED = 1
task.RUNNING = 2
task.CANCELLED = 3
task.FINISHED = 4
task.CLOSING = 5
task.JOB_STATUS (dict)Methodstask=Task(host=None, userid=None, look_for_keys=True, debug_ssh=False)Classe to handle task executionhost (str): Remote hostuserid (str): Remote user idlook_for_keys (bool): Look for keys available in user's .ssh directory
debug_ssh (bool): Open SSH session with debug activated(void) task.load_data_from_file(file_path, mode='json')Loads accumulated task data from external filefile_path (str): Path to filemode (str): Format. Json | Pickle(void) task.save(save_file_path, mode='json', verbose=Falsse)Saves current task status in a external file. Can be used to recover session at a later time.save_file_path (str): Path to filemode (str): Format to use json|pickle.verbose (bool): Print additional information(void) task.set_credentials(credentials, passwd=None):Loads ssh credentials from SSHCredentials object or from a external filecredentials (SSHCredentials|str): SSHCredentials object or a path to a file containing the datapasswd (str): Password to decrypt private key when loaded from file (optional)(void) task.set_private_key(private_path, passwd=None):Inserts private key from external fileprivate_path (str): Path to private key filepasswd (str, optional): Password to decrypt private key(void) task.load_host_config(host_config_path)Loads a pre-defined host configuration file (json format)host_config_path (str): Path to the configuration file(void) task.set_custom_settings(self, ref_setting='default', patch=None, clean=False)Generates a custom queue setting based on existing oneref_setting (str): Base settings to modifypatch (dict): Patch to applyclean (bool): Clean existing settings(void) task.prep_auto_settings(total_cores=0, nodes=0, cpus_per_task=1,  num_gpus=0)Generates queue configuration settings for balancing MPI/OMP/GPU.total_cores (int): Aproximated number of cores to usenodes (int): Number of complete nodes to use (overrides total_cores)cpus_per_task (int): OMP processes per MPI task to allocatenum_gpus (int): Num of GPUs per node to allocate(void) task.set_local_data_bundle(local_data_path, add_files=True)Builds local data bundle from a local directorylocal_data_path (str): Path to local data directoryadd_files (bool): On create, add all files in the directory.(void) task.send_input_data(remote_base_path, overwrite=True, new_only=True)Uploads data bundle files to remote working dirremote_base_path (str): Remote base path for all task activites. Each task will create a unique working dir (re-usable).overwrite (bool): Upload files even if they already exists in the remote working dir.new_only (bool): Overwrite only with newer files(str) task.get_remote_py_script(python_import, files, command, properties='')Generates 1 line python code to be executed in the queue script using python -cpython_import (str): Python import line(s) to include (; separated)files (dict): File names to associate to biobb required path parameterscommand (str): biobb class to launchproperties (dict|str): Either a dict, path to a json or yaml config file or a 1-line Json with the required biobb parameters(str) task.get_remote_comm_line(command, files, use_biobb=False, properties='', cmd_settings=''):Generates a command line for queue script. Can be used to launch a biobb module or any command line remotely.job_name (str): Job name to display (optional, used to identify queue jobs, and stdout/stderr logs)command (str): Command to executefiles (dict): Input/output files. "--" prefix added if only a parameter name is provideduse_biobb (bool): Set to prepend biobb path on hostproperties (dict): BioBB propertiescmd_settings (dict): Additional settings to add to the command line, pre-set bundles can be configured in host config data.(void) task.submit(job_name=None, queue_settings='default', modules=None, local_run_script='', conda_env='', save_file_path=None, poll_time=0)Submits task to remote. Optionally waits until completion.job_name (str): Job name to display in the queuing system. Stdout/stderr logs are named as job.name.(out|err). Optional, defaults to queue default behaviour.queue_settings (str): Label for set of queue settings (defined in host configuration). Use 'custom' for user defined settings (see set_custom_settings)modules (str): modules to activate (defined in host configuration)conda_env (str): Conda environment to activatelocal_run_script (str): Path to local script to run or a string with the script itself (identified by leading '#' tag)save_file_path (str): Path to local task log file to update after submit (Default None),poll_time (int): if set, polls periodically for job completion (seconds)(void) task.cancel(remove_data=False)Cancels running taskremove_data (bool): Removes remoted workign directory.(str) task.check_queue()Check queue status. Returns output of the remote appropriate command(void) check_job(update=True, poll_time=0, save_file_path=None)Prints job status to stdoutupdate (bool): update status before printing itpoll_time (int): poll until job finished. Poll interval in seconds.save_file_path (str): Path to local task log file to update status (Default None),(void) task.get_remote_file(file):Gets file from remote working dirfile (str): File name([stdout, stderr]) task.get_logs()Get queue logs(void) task.get_output_data(local_data_path='', files_only=None, overwrite=True, new_only=True)Downloads remote working dir contents to local pathlocal_data_path (str): Path to local directoryfiles_only ([str]): Only download files in list, if empty download all filesoverwrite (bool): Overwrite local files if they existnew_only (bool): Overwrite only with newer files(void) task.clean_remote()Remove remote working dirslurm.pyTask Class extended to include specific settings for Slurm queueing systemconf/XXX.jsonHost configuration filesUtilitiescredentialsGenerates kay pairs to be consumed by other utilitiescredentials [-h] [--user USERID] [--host HOSTNAME]
            [--pubkey_path PUBKEY_PATH] [--nbits NBITS] --keys_path
            KEYS_PATH [--privkey_path PRIVKEY_PATH]
            {create,get_pubkey,get_private}scp_serviceSimple sftp servicescp_service [-h] --keys_path KEYS_PATH [-i INPUT_FILE_PATH]
                   [-o OUTPUT_FILE_PATH]
                   {get,put,create,file,listdir}ssh_commandSimple remote ssh commandssh_command [-h] --keys_path KEYS_PATH [command [command ...]]slurm_testComplete set of functions to manage slurm submissions remotelyslurm_test [-h] --keys_path KEYS_PATH [--script SCRIPT_PATH]
                  [--local_data LOCAL_DATA_PATH] [--remote REMOTE_PATH]
                  [--queue_settings Q_SETTINGS] [--module MODULE]
                  [--task_data TASK_FILE_PATH]
                  {submit,queue,cancel,status,get_data,put_data}Versionv1.2.2 November 2021Copyright & LicensingThis software has been developed in the MMB group (http://mmb.irbbarcelona.org) at the
BSC (http://www.bsc.es/) & IRB (https://www.irbbarcelona.org/) for the European BioExcel (http://bioexcel.eu/), funded by the European Commission
(EU H2020675728).(c) 2015-2021Barcelona Supercomputing Center(c) 2015-2021Institute for Research in BiomedicineLicensed under theGNU Lesser General Public License v2.1, see the fileLICENSEfor details.