Package: taurus-datamover
Taurus DatamoverPython wrapper for thedatamover toolsthat enable moving data between theZIH fileserverand thetaurus clusterat TU Dresden.To facilitate temporary storage across computing nodes, also provides theWorkspacewrapper for theworkspace toolsBefore you can start working on the taurus cluster, you need to file aproject request. You also need agroup shareon the ZIH fileserver.Getting startedIf you own a project space on taurus and want to transfer files to it, you can askHPC supportto enable a connection to your fileserver.Afterwards, you can useTUD ZIH datamoverto transfer files from the fileserver to the cluster.This python package provides a python wrapper for the datamover tools.UsageDatamoverFirst importtaurus_datamoverand create aDatamoverobjectfromtaurus_datamoverimportDatamover,waitfordm=Datamover()The object will have all datamover commands as methods. The most commonly used methods are:dtlslist directory contents. Equivalent of the the linux commandls.dtcpcopy files likecpdtmvmove files likemvdtrmdelete files likermAll commands take the same arguments like the linux equivalents. Arguments are passed as arguments to the method. For example, the--helpoption tells you how to use them:out,_=dm.dtls('--help').communicate()print(out.decode('utf-8'))returns (truncated):Usage:ls[OPTION]...[FILE]...
ListinformationabouttheFILEs(thecurrentdirectorybydefault).
Sortentriesalphabeticallyifnoneof-cftuvSUXnor--sortisspecified.

Mandatoryargumentstolongoptionsaremandatoryforshortoptionstoo.-a,--alldonotignoreentriesstartingwith.-A,--almost-alldonotlistimplied.and..--authorwith-l,printtheauthorofeachfile[...]The--helpargument is a little special, because it returns immediately. Normally, the commands run on a separate node in the background.For example, listing the contents of the directory/grp/g_biapolproc=dm.dtls('-lh','/grp/g_biapol')initially just prints some messages from the clusters queuing system:srun: job 27549105 queued and waiting for resources(some time passing here, until the job gets assigned to a node...)srun: job 27549105 has been allocated resources
srun: error: ioctl(TIOCGWINSZ): Inappropriate ioctl for device
srun: error: Not using a pseudo-terminal, disregarding --pty optionthe error messages are normal. they are caused by the fact that we did not run the command from an interactive terminal.The whole process takes a while. thereforeDatamoverexecutes the commands in the background and returns immediately, so that you can do other stuff in the background. You can check the status withpoll(). As long as that returnsNone, the process is still in progress:proc.poll()NoneAnd if the process finished it will return the exit code (0 in case of success, an integer larger than 0 in case of an error):proc.poll()0If your code needs to wait for the result, use thewaitforhelper function, which waits for the result and returns the exit code of the command:proc=dm.dtls('-lh','/grp/g_biapol')waitfor(proc)Waiting ..
srun: job 27549647 queued and waiting for resources
......................................................................................................................
srun: job 27549647 has been allocated resources
.
srun: error: ioctl(TIOCGWINSZ): Inappropriate ioctl for device
srun: error: Not using a pseudo-terminal, disregarding --pty option
.
0Only error messages are printed immediately, the normal output of the function is captured and can be retrieved like this:out,_=proc.communicate()print(out.decode('utf-8'))total 12K
drwx------ 20 roha044c 1111111 4.0K Jul  8 13:59 data
drwx------  3 mazo260d 1111111 4.0K Jun 20 11:49 presentations
drwx------  3 johamuel 1111111 4.0K Mar 10 14:53 projectsExamplesRecursively (-r) copy the directorydata/testfrom the fileserver to the project space:waitfor(dm.dtcp('-r','/grp/g_biapol/data/test','/projects/p_bioimage/'))once it is done, we can see (and read) the files on the project space:!ls-la/projects/p_bioimage/test/total 12
drwxrwsr-x 3 tkorten p_bioimage 4096 Aug  3 16:31 .
drwxrws--T 6 root    p_bioimage 4096 Aug  1 13:52 ..
drwx--S--- 2 tkorten p_bioimage 4096 Aug  3 16:31 testHowever, from a normal node, we don't have write access:!rm-r/projects/p_bioimage/test/testrm: cannot remove '/projects/p_bioimage/test/test': Read-only file systemBut using the datamover, we can write to the project space:waitfor(dm.dtrm('-r','/projects/p_bioimage/test'))!ls -la /projects/p_bioimage/test/ls: cannot access '/projects/p_bioimage/test/': No such file or directoryWorkspaceFirst import the workspace-related classesfromtaurus_datamoverimportWorkspace,CacheWorkspacews=Workspace()Workspaceworks very similar to Datamover, except that it provides access to theworkspace command line tools:ws_allocateallocate a new workspacews_listlist existing workspaces for your userws_releaserelease an existing workspace (note all data on that workspace will be deleted within 1 day after releasing the workspace)Let's create a workspace calledmy-workspacethat expires automatically in 30 days:out,_=ws.ws_allocate('my-workspace','30').communicate()workspace_path=out.strip().decode('utf-8')workspace_pathwill look something like/scratch/ws/0/username-my-workspaceList existing workspaces withws_list:out,_=ws.ws_list().communicate()print(out.strip().decode('utf-8'))will return something like:id: my-workspace
     workspace directory  : /scratch/ws/0/username-my-workspace
     remaining time       : 0 days 23 hours
     creation time        : Wed Aug 10 13:16:25 2022
     expiration date      : Thu Aug 11 13:16:25 2022
     filesystem name      : scratch
     available extensions : 10And once we are done, we can release the workspace:ws.ws_release('my-workspace').wait()This hides the workspace and all data will be deleted within 1d after you released the workspace.CacheWorkspaceis a convenience class for creating short-lived temporary workspaces. Instantiating it automatically creates a workspace with the idcachecws=CacheWorkspace()Now you can work with the cache workspace, for example, you can create a temporary folder within:fromtempfileimportTemporaryDirectorytmp=TemporaryDirectory(prefix=cws.name+'/')tmp.name/scratch/ws/0/username-cache/fgixyd8cleaning up the cache (i.e. releasing the cache workspace) is simple:cws.cleanup()If you only need the cache for a single piece of code, you can also use it as a context manager:withCacheWorkspace()ascws:temp_file=cws.name+'/temp.tiff'imsave(temp_file)target_file='/projects/my_project/data.tiff'dm.dtcp(temp_file,target_file)That way, the cache workspace will be created and cleaned up automatically.ContributingContributions are very welcome. Tests can be run withpython tests/test.py.LicenseDistributed under the terms of the BSD-3 license, "biapol-taurus" is free and open source software.SupportIf you need support with the tools listed here, pleaseopen an issue.