Package: nfs4-share
NFSv4-SHAREInstallationUsing PyPi:pip install nfs4_shareOr install from source:clone the repositorycdinto the repository directoryInstall using:pip install .Usage:Runnfs4_share --helpfor arguments. A more detailed description can be foundbelowMotivationTypically Within the high-performance clusters a bulkstorage is exposed as a mount with type Network Filestorage System
version 4 (NFSv4). To protect the data only limited read access is granted and to an even lesser degree write access is
granted. Playing safe can become a problem when the need arises to share the results with other researchers members. Manually
providing and revoking access on an NFSv4 is possible but cumbersome and error prone. Simply copying the data to a shared
location is dangerous and very quickly increases usage of expensive storage. This can be very expensive.The NFSv4-SHARE program is build to solve this problem. It uses properties of the NFSv4 mount to preventdata
duplicationand makes keeping track of permissions relatively easy. Data duplication is prevented by only creatinghard-links to files. Keeping track of permissions is done by
wrapping the hard-linked files in a directory that all share the same permissions.Access to the data and shares itself is controlled byNFSv4 access-control lists (ACLists).
These ACLists consist of entries (ACEntries) which determine what permissions a calling user has. The main differences
between ACLists and the standard POSIX permissions (i.e.rwxrwxrwx) are as follows:multiple users and groups can be definedmore fine-grained permissions can be controlled (13 for files, 14 for directories).A small addition has been made to also control the.htaccessfile of a share to allow data sharing via an apache server.Practical ExampleTake some imaginary source data that is structured as follows:/data/results/
			QC.txt
	/raw_data/sample1/
			run_L1.bam
			run_L2.bam
	/shares/..If you want to do the following:create a share for projectfoobarunder/sharesshare the fileQC.txtfrom directory/data/resultsshare the subdirectorysample1/from directory/raw_dataprovide access to userbobandalicemanage the share with grouppmc_omicsYou run the following command:nfs4_share create /shares/foobar \
	--users bob alice \
	--managing_groups pmc_omics \
	--items /data/results/QC.txt /raw_data/sample1You then end up with a share and source data that is structured as follows:/data/results/
			QC.txt
	/raw_data/sample1/
			run_L1.bam
			run_L2.bam
	/shares/foobar/
			QC.txt
			sample1/
				run_L1.bam
				run_L2.bamUsersbobandalicecould then navigate to the share at/shares/foobarto access the shared data.When they finish or you need to recreate the share, use NFSv4-SHARE to delete the share:nfs4_share delete /shares/foobarImplementation DetailsThe ACLists on the share directory (i.e./shares/foobar) are thede factoshare permissions.Shared FilesWithin the example above, all thefileshave the original ACEntries. These NEED to include reading permissions.Shared DirectoriesAnysubdirectoriesfrom the source that end up in foobar aredifferent subdirectories(!). The directories from
the specified source items have their tree freshly rebuild within the share. For instance, the directory/shares/foobar/sample1is not a hard-link to/raw_data/sample1, but a remake. The directories share the namesample1but have a different
inode number and associated ACList. Within thefoobarshare, the ACList of directorysample1only has the ACEntries
required to have bob and alice read and index files.Tracking changesChanges to a share is tracked with a local git repository. You can use-git {realpath of the local git}to keep track of changes in the share.Unit testsIf the source code is located on an NFSv4 mount with ACLs enabled you can run unit tests as follows:pipinstall.[test]pytest--basetemp=<NFS4_MOUNT>If the source code is not stored on an NFSv4 mount, you should first move it to an NFSv4 mount before unit testing.Luckily, this is already automated in the following script. It will push the source code to the remote server and have the
unittest runs there locally.bashtests/run_test_on_remote_server.sh<ssh_remote_host><remote_working_directory><remote_nfs4_mount_for_creating_shares><test_variables.json>example: bash tests/run_test_on_remote_server.sh gwhorus test_shares_exc /data/isi/p/pmc_research/omics/development/shares tests/variables_UMC.jsonIn case you are working on a cripled OS lacking the "realpath" function then use run_test_on_remote_server_mac.shPython Module InterfaceIf you want to programmatically call this program within python you can use something as follows:fromnfs4_share.manageimportcreate,deletecreate(share_directory="/data/isi/p/pmc_research/omics/shares/share1",domain="op.umcutrecht.nl",items=["file1.txt","file2.txt"])delete(share_directory="/data/isi/p/pmc_research/omics/shares/share1")Upload new version to PyPiThis requires an account atpypi.orgwith access to theproject.Change the version number in./__version__.pyTag a new versiongit tag v0.1.0Install required packages for uploadingpip install --upgrade setuptools wheel twineBuild distpython setup.py sdist bdist_wheelUpload usingtwinetwine upload dist/*