Package: rmsp
Resource Management System (RMS) ManualRMS is an all-rounded python-based workflow management tool for computational analysis. It features a comprehensive logging system with a python API that could be incorporated into jupyter notebook easily (mainly for programmers) and a graphical user interface (mainly for non-programmers). The multiple front-end design allows both non-programmers and programmers to interact using the same system.FeaturesThe entire analysis stored in RMS is basically a single acyclic directed graph with Resources/Fileresources as nodes and Tasks as edges. All nodes have at most one incoming edge but can have unlimited number of outgoing edges. All actions are stored in the database. One can quickly re-fetch a previously generated results.While RMS is originally designed for logging python function, to further extend its use on data analysis, we also provide a set of easy-to-use command line functions.Avoiding duplicated processRMS record most common python functions and objects, and avoid duplicated jobs. Minimal efforts are needed from the users to avoid repeated jobs.Effortless incorporation to existing pipelinesRebuilding an entire analysis into the new system is usually a big hurdle, but not in RMS. RMS provides a simple solution - you can keep using most of your codes in the main body. The only thing to do is to change the import functions.Easy but powerful searchMany systems allow you to find out an upstream file required to generate the results. However, not all systems allow more complicated search. For example, can we search for the results based on some input file 1 and input file 2, processed through either pipeline X with the parameter A or pipeline Y with parameter B? Can we find out peak files using peak calling software J with alignment based on alignment software K? RMS enables these searches so that you can always retrieve your results quicklyEasy Backup / RestoreBackup / Restore is an essential feature to protect a database system from data loss. The core RMS files include all the pipelines used, which are stored in an SQL database. It also allows users to select and backup various files or resource content (For example, only backup the raw file and key result file, and remove any intermediate files). Restoring a database requires.Integrity checkSince RMS uses the absolute path to link to a file, users could easily access the file as for any downstream analysis or visualization. Users can also move or delete any unused intermediate files. Just like Python culture, nothing prevents you from modifying a file if you really do so. We encourage users to be responsible for their own actions. However, RMS will still do a quick check (based on file size) before using the file. An optional integrity check based on MD5 is also available.InstallationTo install Resource Management System, use the following:pipinstallrmspQuick StartTry-it-out for the first timeThis section is designed for lazy people to try RMS without knowing any details. The example codes below can be run without any extra dependency.Create RMS databaseYou only need to run this once when you set up a new database.DBPATH="/path/to/db/"DBNAME="test.db"fromrmspimportrmsutilsrmsutils.create_new_db(f"{DBPATH}{DBNAME}")Connect to RMS database and setup multi-processing runYou need run this every time at the beginning of your scriptDBPATH="/path/to/db/"DBNAME="test.db"DBRESDIR="Resources/"fromrmspimportrmscorerms=rmscore.ResourceManagementSystem(f"{DBPATH}{DBNAME}",f"{DBPATH}{DBRESDIR}")fromrmspimportrmsbuilderrmspool=rmsbuilder.RMSProcessWrapPool(rms,8)rmsb=rmsbuilder.RMSUnrunTasksBuilder(rmspool)fromrmspimportrmsutilsExample of running a functiondefadd_two_numbers(i,j):returni+jadd_two_numbers_pipe=rms.register_pipe(add_two_numbers)result_1_plus_2=add_two_numbers_pipe(1,2)result_pr_plus_4=add_two_numbers_pipe(result_1_plus_2,4)OverviewRMS represents the whole framework using differentRMSEntry.AnRMSEntrycan be in any of the following:PipeTaskResourceFileresourceUnruntaskVirtualresourceAll RMSEntries, exceptunruntaskandvirtualresource, are stored in the database upon creation.UnruntaskandVirtualresourceare only stored in memory. They are not shared across different RMS instances, and they are lost when the RMS instance is close.RMS Entry TypesPipeApipestores all the core information of a python function. Users have to register a python function aspipeso that the running process can be tracked.TaskAtaskstores the information of executing apipe. This includes the pipe ID, the arguments and the execution time.Users cannot register ataskdirectly. Alltasksare generated when running RMS pipes.ResourceAresourceis basically a wrapper to a python object. While you can check the original python object by usingresource.content, you should never modify the content of a resource object. An exception to this is the generator, where it will be consumed instantly after the content is used.Users cannot register aresourcedirectly. Allresourcesare generated when running RMS pipes.FileresourceAFileresourcestores the information of a file. RMS will store the file path of the file using the absolute path (but not real path and relative path). Upon registering a file, RMS will also store the md5 of the file.User can register any file on the file system. Registering the same file path will not create a new entry, unless users specifiedforce=Truewhen registering a file.If a file is generated in certain pipes, it will be automatically registered and linked to the corresponding task. If an existing registered file is overwritten by running thepipe, the oldfileresourceentry will be marked asoverwritten, while a newfileresourcewill be created. If the oldfileresourceis used in running newpipes, an exception will be thrown.Note that RMS do not actively track if files are overwritten by external program. Users can initiate integrity check in RMS routine check using the md5. See rmsutils.UnruntaskAn unruntask is a temporary holder for holding all required information to run a task. Once a task based on this unruntask completes, the unruntask will be automatically removed from RMS. The correspondingtaskcan be found usingunruntask.replacementafter completion.VirtualresourceAvirtualresourceis a temporary holder for anyresourceorfileresourceobjects to be generated when running the pipes. Once a task leading to thevirtualresourcecompletes, thevirtualresourcewill be automatically removed from RMS. The correspondingresourceorfileresourcecan be found usingvirtualresource.replacementafter completion.Shared RMS Entry PropertiesDescriptionThedescriptionof an RMSEntry is a field used by users to annotate.infoTheinfoof an RMSEntry is a field used by RMS to indicate different status. This includes:overwritten: If a task overwrites an already registered file, the RMSEntry representing the old file will have a statusoverwritten. Also, an RMS Entry will be created to represent the new file. In the database, there could be multiple RMSEntry pointing to the same path but there must be at most one RMSEntry without theoverwrittenstate.obsolete: Upon changing any input of a task, the task and all downstream RMSEntry will be marked asobsoletebecause these entries are no longer up-to-date. Only if these RMSEntry is updated, the obsolete status will be removed. See also the substitution section.sourcecode (Pipe only): If a function registered as Pipe is defined in__main__, the source code of the function will be stored as sourcecode.outputfunc_sourcecode (Pipe only): Similar to sourcecode, but this is the source code for output_func.User Guide - Programmatic accessThe user guide here targets advanced programmers who want to gain full access to all features of RMS. If you are not familiar with programming but want to use little amount of programming to do repetitive jobs, please see the section User Guide - GUI.InitializationCreating the DBRMS stores all the RMS entries in an sqlite3 database. You need to create a new database the first time you use RMS. Example codes are shown below:DBPATH="/path/to/db/"DBNAME="test.db"fromrmspimportrmsutilsrmsutils.create_new_db(f"{DBPATH}{DBNAME}")Here "test.db" will be created in your destinated folderConnecting to the DBEvery time you use RMS, you need to initialize RMS with the following commands:DBPATH="/path/to/db/"DBNAME="test.db"DBRESDIR="Resources/"fromrmspimportrmscorerms=rmscore.ResourceManagementSystem(f"{DBPATH}{DBNAME}",f"{DBPATH}{DBRESDIR}")The DBRESDIR is used to store the user-dumpedResourcecontent.Setting up multiprocessing supportIn addition to the normal connection steps, you will need to add a few lines to create multiprocessing support for RMS as well.DBPATH="/path/to/db/"DBNAME="test.db"DBRESDIR="Resources/"fromrmspimportrmscorerms=rmscore.ResourceManagementSystem(f"{DBPATH}{DBNAME}",f"{DBPATH}{DBRESDIR}")fromrmspimportrmsbuilderrmspool=rmsbuilder.RMSProcessWrapPool(rms,8)rmsb=rmsbuilder.RMSUnrunTasksBuilder(rmspool)Starting the GUI(GUI may not be available at beta version)You can initialize the GUI in the same console.fromrmspimportrmsguirmsgui.start_GUI(rms)For jupyter user, you need to run the following prior to starting a GUI:%guiqt5X11 may be required for running the GUI.Registering functions asPipesBefore running anything in RMS, you need to register a python function first. After a successful registration, RMS will create aPipeentry in the database. There are several important parameters when registering a function to RMS.Simple functionIn the simplest case, one can just register the function with default parameter. Bound method is currently not supported.defadd_two_numbers(i,j):returni+jadd_two_numbers_pipe=rms.register_pipe(add_two_numbers)Function from a packageimportbiodata.bedread_all_pipe=rms.register_pipe(biodata.bed.BED3Reader.read_all)GeneratorFor generator function in python, one should setreturn_volatileasTrue. The returnedResourcefrom this task will be marked as volatile automatically.defgenerate_number(n):foriinrange(n):yieldigenerate_number_pipe=rms.register_pipe(generate_number,return_volatile=True)Functions with output filesFor function that outputs a file, we must add anoutput functionto indicate what output files are expected. The output function should have the same arguments as the original function, and should return a list of output files. Note that the order of output files could be important, when one use existing tasks to generate templates. See also: (TO-BE-ADDED). After the pipe has complete, all output files indicated from the output function will be automatically registered asFileResources.defwrite_hello_world(output_file1,output_file2):withopen(output_file1,"w")asf:f.write("Hello World 1!")withopen(output_file2,"w")asf:f.write("Hello World 2!")defOUTPUT_write_hello_world(output_file1,output_file2):return[output_file1,output_file2]write_hello_world_pipe=rms.register_pipe(write_hello_world,output_func=OUTPUT_write_hello_world)Function with unknown behaviorFor function with returned values or output files  that cannot be repeated by running the function with the same parameter, one should set the parameteris_deterministicasFalse.defget_random_number():importrandomreturnrandom.random()get_random_number_pipe=rms.register_pipe(get_random_number,is_deterministic=False)We also recommend settingis_deterministicasFalsefor functions that involve retrieving files from the internet (such asurlretrieve).Registering a function multiple timesIf you register the same function using the same parameters, the already registered pipe is returned instead of creating a new pipe entry. This does not hold, however, when you are registering function defined in__main__. While RMS is trying hard to make the same function defined in different__main__the same, we still do not guarantee to find a matching old pipe.Independent to other functionsA custom registered function must not depend on any function.Importing aPipefrom existing libraryImportingpipeis similar to importing normal python function. To simplify the conversion, you can convert normal python imports into RMS imports using rmsutils. Note that you MUST register the library function first.Example of RMS import:GenomicCollection=rms.rmsimport(moduleobj='genomictools',varname='GenomicCollection')bb=rms.rmsimport(moduleobj='biodata.bed',return_first_module=False)The rmsutils provide the following function to convert normal python imports into the aboveprint(rmsutils.convert_import_str('''# Normal python importfrom genomictools import GenomicCollectionimport biodata.bed as bb'''))Note that the following does not work:frombiodata.bedimport*We previously made this import viable. However, this may create serious security issue and we decided to remove such features.Running aPipeAfter a function is registered, there are two methods to run the pipe. No matter which method is used, upon successful completion of the pipe, atask, all outputresourcesandfileresourcesentries will be automatically registered to the database.Conventional wayThe conventional way, resembling normal python function calling, is to call the pipe directly with arguments. The command will block until the Pipe has finished.result_1_plus_2=add_two_numbers_pipe(1,2)result_pr_plus_4=add_two_numbers_pipe(result_1_plus_2,4)The following codes also work exactly the same.result_1_plus_2=rms.run(add_two_numbers_pipe,args=[1,2])result_pr_plus_4=rms.run(add_two_numbers_pipe,args=[result_1_plus_2,4])Now theresult_1_plus_2is aResourcethat stores the content of the task.result_1_plus_2.content# 3As you may have noticed, you can also useResourceobject directly on running other pipes.result_pr_plus_4.content# 7Running Pipe using RMSBuilderCreate a virtual RMS interface using RMSBuilder. Most commands are similar to running a task using the conventional way. The key difference of using a builder is that when users run the pipe, a list ofunruntasksandvirtualresourcesare created rather than executing the pipes instantly.First create the builder.fromrmspimportrmsbuilderrmspool=rmsbuilder.RMSProcessWrapPool(rms,8)rmsb=rmsbuilder.RMSUnrunTasksBuilder(rmspool)Theadd_two_numbers_builder_pipe=rmsb.register_pipe(add_two_numbers)result_1_plus_2_vr=add_two_numbers_builder_pipe(1,2)result_3_plus_4_vr=add_two_numbers_builder_pipe(3,4)result_r1_plus_r2_vr=add_two_numbers_builder_pipe(result_1_plus_2_vr,result_3_plus_4_vr)# All unruntasks are on hold until explict call to execution.To start the execution, users need to callexecute_builder. Theseunruntaskswill then be run in parallel in a pool.rmsb.execute_builder()To access the result after completion, one should useresult_r1_plus_r2=result_r1_plus_r2_vr.replacementresult_r1_plus_r2.contentRegistering FileResourceTo keep track of the files in RMS, all input files should be registered as aFileResource. For any input function, if you want to use a file parameter, use eitherrms.register_fileorrms.file_from_pathon the file path.An example of function that copy a filedefcopy_a_file(input_file,output_file):importshutil# Do some other fancy stuff in this functionshutil.copy2(input_file,output_file)defOUTPUT_copy_a_file(input_file,output_file):return[output_file]copy_a_file_pipe=rms.register_pipe(copy_a_file,output_func=OUTPUT_copy_a_file)For files that are not generated through RMS, (e.g. downloaded from other sources) one can register.# Suppose the input file was obtained from somewhere elseinput_rmsfile_object=rms.register_file('/path/to/source/file')intermediate_file_path='/path/to/intermediate/file'copy_a_file_pipe(input_rmsfile_object,intermediate_file_path)# Since the intermediate file was generated in RMS, we only need to use file_from_pathintermediate_file_object=rms.file_from_path(intermediate_file_path)output_file_path='/path/to/output/file'copy_a_file_pipe(input_rmsfile_object,output_file_path)In the following example, although you can still get the desired result, the input file path is mistaken as a string parameter instead of a file parameter.# An incorrect exampleinput_file_path='/path/to/source/file'intermediate_file_path='/path/to/intermediate/file'copy_a_file_pipe(input_file_path,intermediate_file_path)Finding RMS entryThis is one of the key advantage for programmer to use RMS - they can search their RMS entry with ease.Here we demonstrate several examples to find the related RMS Entries.(The example will be shown later)RerunningHuman error is unavoidable. There are always cases that a wrong file is used or a wrong parameter is set. Therefore, it is better to have a system to let users rerun the task with the wrong files.You should mark the wrong files with "deprecated status".RMS utilitiesCreation of new databaseDB_PATH="/path/to/db/test.db"fromrmspimportrmsutilsrmsutils.create_new_db(DB_PATH)ExportsBackup and RestoreBackupTo backup all the files, you can use the rmsutilsfromrmspimportrmsutilsrmsutils.backup('target_path','a.db','resource_dump_dir')RestoreThe files will be placed at the same path as before. However,fromrmspimportrmsutilsrmsutils.restore('src_path','target_path')User Guide - GUIThe GUI is useful to visualize all your tasks, files and resources. It allows instant analysis to certain variables. The template system also allows you to run simple analysis if you are unfamiliar with coding.  Please refer to rmsp-gui for detailsFAQ / Important NotesThis section includes some of the important notes that one should keep in mind when using RMS.RMS does not store any information about the environment. It does not keep track of any external executables. Rather, it basically relies on the conda environment. Hence when doing the backup / restore, make sure you also backup the environment yourself.