Package: evercas
EverCasEverCas is a content-addressable file management system. What does that
mean? Simply, that EverCas manages a directory where files are saved
based on the file's hash.Typical use cases for this kind of system are ones where:Files are written once and never change (e.g. image storage).It's desirable to have no duplicate files (e.g. user uploads).File metadata is stored elsewhere (e.g. in a database).FeaturesFiles are stored once and never duplicated.Uses an efficient folder structure optimized for a large number of
files. File paths are based on the content hash and are nested based
on the firstnnumber of characters.Can save files from local file paths or readable objects (open file
handlers, IO buffers, etc).Pluggable put strategies, allowing fine-grained control of how files
are added.Able to repair the root folder by reindexing all files. Useful if
the hashing algorithm or folder structure options change or to
initialize existing files.Supports any hashing algorithm available viahashlib.new.Python 3.10+ compatible.Support for hard-linking files into the EverCas-managed directory on
compatible filesystemsLinksProject:https://github.com/weedonandscott/evercasDocumentation:https://weedonandscott.github.io/evercas/PyPI:https://pypi.python.org/pypi/evercas/QuickstartInstall using pip:pip install evercasInitializationfromevercasimportEverCasDesignate a root folder forEverCas. If the folder doesn't already
exist, it will be created.# Set the `depth` to the number of subfolders the file's hash should be split when saving.# Set the `width` to the desired width of each subfolder.fs=EverCas('temp_evercas',depth=4,width=1,algorithm='sha256')# With depth=4 and width=1, files will be saved in the following pattern:# temp_evercas/a/b/c/d/efghijklmnopqrstuvwxyz# With depth=3 and width=2, files will be saved in the following pattern:# temp_evercas/ab/cd/ef/ghijklmnopqrstuvwxyzNOTE:Thealgorithmvalue should be a valid string argument tohashlib.new().Basic UsageEverCassupports basic file storage, retrieval, and removal as well as
some more advanced features like file repair.Storing ContentAdd content to the folder using either readable objects (e.g.StringIO) or file paths (e.g.'a/path/to/some/file').fromioimportStringIOsome_content=StringIO('some content')address=fs.put(some_content)# Or if you'd like to save the file with an extension...address=fs.put(some_content,'.txt')# Put all files in a directoryforsrcpath,addressinfs.putdir("dir"):#...# Put all files in a directory tree recursivelyforsrcpath,addressinfs.putdir("dir",recursive=True):#...# Put all files in a directory tree using same extensionsforsrcpath,addressinfs.putdir("dir",extensions=True):# address.abspath will have same file extension as srcpath# The id of the file (i.e. the hexdigest of its contents).address.id# The absolute path where the file was saved.address.abspath# The path relative to fs.root.address.relpath# Whether the file previously existed.address.is_duplicateRetrieving File AddressGet a file'sHashAddressby address ID or path. This address would be
identical to the address returned byput().assertfs.get(address.id)==addressassertfs.get(address.relpath)==addressassertfs.get(address.abspath)==addressassertfs.get('invalid')isNoneRetrieving ContentGet aBufferedReaderhandler for an existing file by address ID or
path.fileio=fs.open(address.id)# Or using the full path...fileio=fs.open(address.abspath)# Or using a path relative to fs.rootfileio=fs.open(address.relpath)NOTE:When getting a file that was saved with an extension, it's
not necessary to supply the extension. Extensions are ignored when
looking for a file based on the ID or path.Removing ContentDelete a file by address ID or path.fs.delete(address.id)fs.delete(address.abspath)fs.delete(address.relpath)NOTE:When a file is deleted, any parent directories above the file
will also be deleted if they are empty directories.Advanced UsageBelow are some of the more advanced features ofEverCas.Repairing FilesTheEverCasfiles may not always be in sync with it'sdepth,width, oralgorithmsettings (e.g. ifEverCastakes ownership of a
directory that wasn't previously stored using content hashes or if theEverCassettings change). These files can be easily reindexed usingrepair().repaired=fs.repair()# Or if you want to drop file extensions...repaired=fs.repair(extensions=False)WARNING:It's recommended that a backup of the directory be made
before repairing just in case something goes wrong.Walking Corrupted FilesInstead of actually repairing the files, you can iterate over them for
custom processing.forcorrupted_path,expected_addressinfs.corrupted():# do somethingWARNING:EverCas.corrupted()is a generator so be aware that
modifying the file system while iterating could have unexpected results.Walking All FilesIterate over files.forfileinfs.files():# do something# Or using the class' iter method...forfileinfs:# do somethingIterate over folders that contain files (i.e. ignore the nested
subfolders that only contain folders).forfolderinfs.folders():# do somethingComputing SizeCompute the size in bytes of all files in therootdirectory.total_bytes=fs.size()Count the total number of files.total_files=fs.count()# Or via len()...total_files=len(fs)Hard-linking filesYou can use the built-in "link" put strategy to hard-link files into
the EverCas directory if the platform and filesystem support it. This
will automatically and silently fall back to copying if a hard-link
can't be made, e.g. because the source is on a different device, the
EverCas directory is on a filesystem that does not support hard links or
the source file already has the operating system's maximum allowed
number of hard links to it.newpath=fs.put("file/path",put_strategy="link").abspathassertos.path.samefile("file/path",newpath)Custom Put StrategyFine-grained control over how each file or file-like object is stored in
the underlying filesytem.# Implement your own put strategydefmy_put_strategy(evercas,src_stream,dst_path):# src_stream is the source data to insert# it is a EverCas.Stream object, which is a Python file-like object# Stream objects also expose the filesystem path of the underlying# file via the src_stream.name property# dst_path is the path generated by EverCas, based on the hash of the# source data# src_stream.name will be None if there is not an underlying file path# available (e.g. a StringIO was passed or some other non-file# file-like)# Its recommended to check name property is available before usingifsrc_stream.name:# Example: rename files instead of copying# (be careful with underlying file paths, make sure to test your# implementation before using it).os.rename(src_stream.name,dst_path)# You can also access properties and methods of the EverCas instance# using the evercas parameteros.chmod(dst_path,EverCas.fmode)else:# The default put strategy is available for use as# PutStrategies.copy# You can manually call other strategies if you want fallbacks# (recommended)PutStrategies.copy(EverCas,src_stream,dst_path)# And use it like:fs.put("myfile",put_strategy=my_put_strategy)For more details, please see the full documentation athttps://weedonandscott.github.io/evercas/.AcknowledgementsThis software is based on HashFS, made by @dgilland with @x11x contributions, and inspired by parts of dud, by @kevin-hanselman.