Package: shelephant
shelephantCommand-line arguments with a memory (stored in YAML-files).Documentation:https://shelephant.readthedocs.ioContentsOverviewHallmark feature: Copy with restartCommand-line toolsFile informationFile operationsYAML file operationsDisclaimerGetting shelephantUsing condaUsing PyPiFrom sourceDetailed examplesGet files from remote, allowing restartsAvoid recomputing checksumsSend files to hostBasic copyRestartOverviewHallmark feature: Copy with restartshelephantpresents you with a way to copy files (from a remote, using SSH) in two steps:Collect a list of files that should be copied in a YAML-file,
allowing you toreview and customisethe copy operation
(e.g. bychanging the orderand making last-minute manual changes).Perform the copy, efficiently skipping files that are identical.Typical workflow:# Collect files to copy & compute their checksum (e.g. on remote system)# - creates "shelephant_dump.yaml"shelephant_dump*.hdf5# - reads "shelephant_dump.yaml"# - creates "shelephant_checksum.yaml"shelephant_checksum# Combine all needed info (locally)# - reads "shelephant_dump.yaml" and "shelephant_checksum.yaml"# - creates "shelephant_hostinfo.yaml"shelephant_hostinfo--hostmyhost--prefix/some/path--files--checksum# Copy from remote (can be restarted and any time, existing files are skipped)# - reads "shelephant_hostinfo.yaml"shelephant_getThe filenames can be customised.To copytoa remote system useshelephant_send.Get details in the help of the respective commands, e.g.shelephant_dump --help.shelephantworks for both local as remote copy actions.Command-line toolsFile informationshelephant_dump: list filenames in a YAML file.shelephant_checksum: get the checksums of files listed in a YAML file.shelephant_hostinfo: collect host information (from a remote system).File operationsshelephant_get: copy from remote, based on earlier stored information.shelephant_send: copy to remote, based on earlier stored information.shelephant_rm: remove files listed in a YAML file.shelephant_cp: copy files listed in a YAML file.shelephant_mv: move files listed in a YAML file.YAML file operationsshelephant_extract: isolate a (number of) field(s) in a (new) YAML file.shelephant_merge: merge two YAML-files.shelephant_parse: parse a YAML-files and print to screen.DisclaimerThis library is free to use under theMIT license. Any additions are very much appreciated, in terms of suggested functionality, code, documentation, testimonials, word-of-mouth advertisement, etc. Bug reports or feature requests can be filed onGitHub. As always, the code comes with no guarantee. None of the developers can be held responsible for possible mistakes.Download:.zip file|.tar.gz file.(c -MIT) T.W.J. de Geus (Tom) |tom@geus.me|www.geus.me|github.com/tdegeus/shelephantGetting shelephantUsing condacondainstall-cconda-forgeshelephantThis will also download and install all necessary dependencies.Using PyPipipinstallshelephantThis will also download and install the necessary Python modules.From source# Download shelephantgitcheckouthttps://github.com/tdegeus/shelephant.gitcdshelephant# Installpython-mpipinstall.This will also download and install the necessary Python modules.Detailed examplesGet files from remote, allowing restartsSuppose that we want to copy all*.txtfiles
from a certain directory/path/where/files/are/storedon a remote hosthostname.First step, collect informationon the host:# connect to the hostsshhostname# go the relevant location at the hostcd"/path/where/files/are/stored/on/remote"# list files to copyshelephant_dump-ofiles_to_copy.yaml*.txt# optional but useful, get the checksum of the files to copyshelephant_checksum-ofiles_checksum.yamlfiles_to_copy.yaml# disconnectexit# or press Ctrl + DSecond step, copy files to thelocal system, collecting everything in a single place:# go to the relevant location on the local system# (often this is new directory)cd"/path/where/to/copy/to"# get the file-information compiled on the host# and store in a (temporary) local file# note that all paths are on the remote system,# and that they are now copied using secure-copy (scp)shelephant_hostinfo\-oremote_info.yaml\--host"hostname"\--prefix"/path/where/files/are/stored/on/remote"\--files"files_to_copy.yaml "\--checksum"files_checksum.yaml"# finally, get the files using secure copy# (the files are stored relative to the path of 'remote_info.yaml',# identically to how they are relative to 'files_to_copy.yaml' on remote)shelephant_getremote_info.yamlIf you use the default filenames forshelephant_dump(shelephant_dump.yaml) andshelephant_checksum(shelephant_checksum.yaml) remotely,
you can also specify--filesand--checksumwithout an argument.An interesting benefit that derives from having computed the checksums on the host,
is thatshelephant_getcan be stopped and restarted:only files that do not exist locally, or that were only partially copied
(whose checksum does not match the remotely computed checksum), will be copied;
all fully copied files will be skipped.Let's further illustrate with a complete example. On the host, suppose that we have/path/where/files/are/stored/on/remote
- foo.txt
- bar.txtThis will give,files_to_copy.yaml:-foo.txt-bar.txtfiles_checksum.yaml(for example):-2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae-fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9This information will be collected toremote_info.yamlhost: hostname
root: /path/where/files/are/stored/on/remote
files:
    - foo.txt
    - bar.txt
checksum:
    - 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
    - fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9shelephant_getwill now copyfoo.txtandbar.txtrelative to the directory ofremote_info.yaml(in this case in the same folder asremote_info.yaml).
It will skip any files whose filename and checksum match to target ones.Avoid recomputing checksumsSuppose that we want to restart multiple times, or that we
update the files present on the remote after copying them initially.
In that case, we can use previously computed
checksums to avoid recomputing them
(which can be costly for large files).First step, update informationon the host:# connect to the hostsshhostname# go the relevant location at the hostcd"/path/where/files/are/stored/on/remote"# collect the previously computed informationshelephant_hostinfo-oprecomputed_checksums.yaml-ffiles_to_copy.yaml-cfiles_checksum.yaml# list files to copyshelephant_dump-ofiles_to_copy.yaml*.txt# get the checksum of the files to copy, where possible reading precomputed valuesshelephant_checksum-ofiles_checksum.yamlfiles_to_copy.yaml-lprecomputed_checksums.yaml# disconnectexit# or press Ctrl + DSecond step, copy files to thelocal system, collecting everything in a single place:# go to the relevant location on the local system# (often this is new directory)cd"/path/where/to/copy/to"# collect the previously computed informationshelephant_hostinfo-oprecomputed_checksums.yaml-ffiles_present.yaml-cfiles_checksum.yaml# list files currently present locallyshelephant_dump-ofiles_present.yaml*.txt# get the checksum of the files to copy, where possible reading precomputed valuesshelephant_checksum-ofiles_checksum.yamlfiles_present.yaml-lprecomputed_checksums.yaml# combine local files and checksumsshelephant_hostinfo-oprecomputed_checksums.yaml-ffiles_present.yaml-cfiles_checksum.yaml# get the file-information compiled on the host [as before]shelephant_hostinfo\-oremote_info.yaml\--host"hostname"\--prefix"/path/where/files/are/stored/on/remote"\--files"files_to_copy.yaml "\--checksum"files_checksum.yaml"# get the files using secure copy# use the precomputed checksums instead of computing themshelephant_getremote_info.yaml--local"precomputed_checksums.yaml"Send files to hostBasic copySuppose that we want to copy all*.txtfiles
from a certain local directory/path/where/files/are/stored/locally,
to a remote hosthostname.First, we will collect informationlocally:# go the relevant location (locally)cd/path/where/files/are/stored/locally# list files to copyshelephant_dump-ofiles_to_copy.yaml*.txtThen, we will specify some basic information about the host# specify basic information about the host# and store in a (temporary) local fileshelephant_hostinfo\-oremote_info.yaml\--host"hostname"\--prefix"/path/where/to/copy/to/on/remote"\Now we can copy the files:shelephant_sendfiles_to_copy.yamlremote_info.yamlRestartSuppose that copying was interrupted before completing.
We can avoid recopying by again using the checksums.
We therefore need to know which files are already present remotely
and which checksum they have.
Thereto:# connect to the hostsshhostname# go the relevant location at the hostcd"/path/where/to/copy/to/on/remote"# list files to copyshelephant_dump-ofiles_to_copy.yaml*.txt# get the checksum of the files to copyshelephant_checksum-ofiles_checksum.yamlfiles_to_copy.yaml# disconnectexit# or press Ctrl + DNow we will complement the basic host-info:shelephant_hostinfo\-oremote_info.yaml\--host"hostname"\--prefix"/path/where/to/copy/to/on/remote"\--files"files_to_copy.yaml "\--checksum"files_checksum.yaml"And restart the partial copy:shelephant_sendfiles_to_copy.yamlremote_info.yaml