Package: cs.hashindex
A command and utility functions for making listings of file content hashcodes
and manipulating directory trees based on such a hash index.Latest release 20240216:HashIndexCommand.cmdlinkto,cmd_rearrange: run the link/mv stuff with sys.stdout in line buffered mode.DO not get hashcodes from symlinks.HashIndexCommand.cmd_ls: ignore None hashcodes, do not set xit=1.New run_remote_hashindex() and read_remote_hashindex() functions.dir_filepaths: skip dot files, the fstags .fstags file and nonregular files.Functiondir_filepaths(dirpath: str, *, fstags: cs.fstags.FSTags)Generator yielding the filesystem paths of the files indirpath.Functiondir_remap(srcdirpath: str, fspaths_by_hashcode: Mapping[cs.hashutils.BaseHashCode, List[str]], *, hashname: str)Generator yielding(srcpath,[remapped_paths])2-tuples
based on the hashcodes keyingrfspaths_by_hashcode.Functionfile_checksum(fspath: str, hashname: str = 'sha256', *, fstags: cs.fstags.FSTags) -> Optional[cs.hashutils.BaseHashCode]Return the hashcode for the contents of the file atfspath.
Warn and returnNoneonOSError.Functionget_fstags_hashcode(fspath: str, hashname: str, fstags: cs.fstags.FSTags) -> Tuple[Optional[cs.hashutils.BaseHashCode], Optional[os.stat_result]]Obtain the hashcode cached in the fstags if still valid.
Return a 2-tuple of(hashcode,stat_result)wherehashcodeis aBaseHashCodesubclass instance is valid
orNoneif missing or no longer valid
andstat_resultis the currentos.statresult forfspath.Functionhashindex(fspath, *, hashname: str, fstags: cs.fstags.FSTags)Generator yielding(hashcode,filepath)2-tuples
for the files infspath, which may be a file or directory path.
Note that it yields(None,filepath)for files which cannot be accessed.ClassHashIndexCommand(cs.cmdutils.BaseCommand)Tool to generate indices of file content hashcodes
and to link files to destinations based on their hashcode.Command line usage:Usage: hashindex subcommand...
    Generate or process file content hash listings.
  Subcommands:
    help [-l] [subcommand-names...]
      Print help for subcommands.
      This outputs the full help for the named subcommands,
      or the short help for all subcommands if no names are specified.
      -l  Long help even if no subcommand-names provided.
    linkto [-f] [-h hashname] [--mv] [-n] [-q] [-s] srcdir dstdir < hashindex
      Link files from srcdir to dstdir according the input hash index.
      -f    Force: link even if the target already exists.
      -h hashname
            Specify the hash algorithm, default: sha256
      --mv  Move: unlink the original after a successful hard link.
      -n    No action; recite planned actions.
      -q    Quiet. Do not report actions.
      -s    Symlink the source file instead of hard linking.
    ls [-h hashname] [-r] paths...
      Walk the filesystem paths and emit a listing.
      -h hahsname   Specify the file content hash alogrith name.
      -r            Emit relative paths in the listing.
                    This requires each path to be a directory.
    rearrange [options...] {[[user@]host:]refdir|-} [[user@]rhost:]targetdir
      Rearrange files in targetdir based on their positions in refdir.
      Options:
        -e ssh_exe  Specify the ssh executable.
        -h hashname Specify the file content hash algorithm name.
        -H hashindex_exe
                    Specify the remote hashindex executable.
        --mv        Move mode.
        -n          No action, dry run.
        -s          Symlink mode.
    shell
      Run a command prompt via cmd.Cmd using this command's subcommands.Functionmain(argv=None)Commandline implementation.Functionmerge(srcpath: str, dstpath: str, *, opname=None, hashname: str, move_mode: bool = False, symlink_mode=False, doit=False, quiet=False, fstags: cs.fstags.FSTags)Mergesrcpathtodstpath.Ifdstpathdoes not exist, move/link/symlinksrcpathtodstpath.
Otherwise checksum their contents and raiseFileExistsErrorif they differ.Functionpaths_remap(srcpaths: Iterable[str], fspaths_by_hashcode: Mapping[cs.hashutils.BaseHashCode, List[str]], *, hashname: str)Generator yielding(srcpath,fspaths)2-tuples.Functionread_hashindex(f, start=1, *, hashname: str)A generator which reads line from the filefand yields(hashcode,fspath)2-tuples.
If there are parse errors thehashcodeorfspathmay beNone.Functionread_remote_hashindex(rhost: str, rdirpath: str, *, hashname: str, ssh_exe=None, hashindex_exe=None, check=True)A generator which reads a hashindex of a remote directory,
This runs:hashindex ls -h hashname -r rdirpathon the remote host.
It yields(hashcode,fspath)2-tuples.Parameters:rhost: the remote host, oruser@hostrdirpath: the remote directory pathhashname: the file content hash algorithm namessh_exe: thesshexecutable,
defaultDEFAULT_SSH_EXE:'ssh'hashindex_exe: the remotehashindexexecutable,
defaultDEFAULT_HASHINDEX_EXE:'hashindex'check: whether to check that the remote command has a0return code,
defaultTrueFunctionrearrange(srcdirpath: str, rfspaths_by_hashcode, *, hashname: str, move_mode: bool = False, symlink_mode=False, doit: bool, quiet: bool = False, fstags: cs.fstags.FSTags, runstate: cs.resources.RunState)Rearrange the files indirpathaccording to the
hashcode->[relpaths]fspaths_by_hashcode.Functionrun_remote_hashindex(rhost: str, argv, *, ssh_exe=None, hashindex_exe=None, check: bool = True, doit: bool = True, **subp_options)Run a remotehashindexcommand.
Return theCompletedProcessresult orNoneifdoitis false.
Note that as withcs.psutils.run, the arguments are resolved
viacs.psutils.prep_argv.Parameters:rhost: the remote host, oruser@hostargv: the command line arguments to be passed to the
remotehashindexcommandssh_exe: thesshexecutable,
defaultDEFAULT_SSH_EXE:'ssh'hashindex_exe: the remotehashindexexecutable,
defaultDEFAULT_HASHINDEX_EXE:'hashindex'check: whether to check that the remote command has a0return code,
defaultTruedoit: whether to actually run the command, defaultTrueOther keyword parameters are passed therough tocs.psutils.run.Functionset_fstags_hashcode(fspath: str, hashcode, S: os.stat_result, fstags: cs.fstags.FSTags)Recordhashcodeagainstfspath.Release LogRelease 20240216:HashIndexCommand.cmdlinkto,cmd_rearrange: run the link/mv stuff with sys.stdout in line buffered mode.DO not get hashcodes from symlinks.HashIndexCommand.cmd_ls: ignore None hashcodes, do not set xit=1.New run_remote_hashindex() and read_remote_hashindex() functions.dir_filepaths: skip dot files, the fstags .fstags file and nonregular files.Release 20240211.1:
Better module docstring.Release 20240211:
Initial PyPI release: "hashindex" command and utility functions for listing file hashcodes and rearranging trees based on a hash index.