Package: hashfs
HashFS is a content-addressable file management system. What does that mean? Simply, that HashFS manages a directory where files are saved based on the file’s hash.Typical use cases for this kind of system are ones where:Files are written once and never change (e.g. image storage).It’s desirable to have no duplicate files (e.g. user uploads).File metadata is stored elsewhere (e.g. in a database).FeaturesFiles are stored once and never duplicated.Uses an efficient folder structure optimized for a large number of files. File paths are based on the content hash and are nested based on the firstnnumber of characters.Can save files from local file paths or readable objects (open file handlers, IO buffers, etc).Able to repair the root folder by reindexing all files. Useful if the hashing algorithm or folder structure options change or to initialize existing files.Supports any hashing algorithm available viahashlib.new.Python 2.7+/3.3+ compatible.LinksProject:https://github.com/dgilland/hashfsDocumentation:http://hashfs.readthedocs.orgPyPI:https://pypi.python.org/pypi/hashfs/TravisCI:https://travis-ci.org/dgilland/hashfsQuickstartInstall using pip:pip install hashfsInitializationfromhashfsimportHashFSDesignate a root folder forHashFS. If the folder doesn’t already exist, it will be created.# Set the `depth` to the number of subfolders the file's hash should be split when saving.# Set the `width` to the desired width of each subfolder.fs=HashFS('temp_hashfs',depth=4,width=1,algorithm='sha256')# With depth=4 and width=1, files will be saved in the following pattern:# temp_hashfs/a/b/c/d/efghijklmnopqrstuvwxyz# With depth=3 and width=2, files will be saved in the following pattern:# temp_hashfs/ab/cd/ef/ghijklmnopqrstuvwxyzNOTE:Thealgorithmvalue should be a valid string argument tohashlib.new().Basic UsageHashFSsupports basic file storage, retrieval, and removal as well as some more advanced features like file repair.Storing ContentAdd content to the folder using either readable objects (e.g.StringIO) or file paths (e.g.'a/path/to/some/file').fromioimportStringIOsome_content=StringIO('some content')address=fs.put(some_content)# Or if you'd like to save the file with an extension...address=fs.put(some_content,'.txt')# The id of the file (i.e. the hexdigest of its contents).address.id# The absolute path where the file was saved.address.abspath# The path relative to fs.root.address.relpath# Whether the file previously existed.address.is_duplicateRetrieving File AddressGet a file’sHashAddressby address ID or path. This address would be identical to the address returned byput().assertfs.get(address.id)==addressassertfs.get(address.relpath)==addressassertfs.get(address.abspath)==addressassertfs.get('invalid')isNoneRetrieving ContentGet aBufferedReaderhandler for an existing file by address ID or path.fileio=fs.open(address.id)# Or using the full path...fileio=fs.open(address.abspath)# Or using a path relative to fs.rootfileio=fs.open(address.relpath)NOTE:When getting a file that was saved with an extension, it’s not necessary to supply the extension. Extensions are ignored when looking for a file based on the ID or path.Removing ContentDelete a file by address ID or path.fs.delete(address.id)fs.delete(address.abspath)fs.delete(address.relpath)NOTE:When a file is deleted, any parent directories above the file will also be deleted if they are empty directories.Advanced UsageBelow are some of the more advanced features ofHashFS.Repairing FilesTheHashFSfiles may not always be in sync with it’sdepth,width, oralgorithmsettings (e.g. ifHashFStakes ownership of a directory that wasn’t previously stored using content hashes or if theHashFSsettings change). These files can be easily reindexed usingrepair().repaired=fs.repair()# Or if you want to drop file extensions...repaired=fs.repair(extensions=False)WARNING:It’s recommended that a backup of the directory be made before repairing just in case something goes wrong.Walking Corrupted FilesInstead of actually repairing the files, you can iterate over them for custom processing.forcorrupted_path,expected_addressinfs.corrupted():# do somethingWARNING:HashFS.corrupted()is a generator so be aware that modifying the file system while iterating could have unexpected results.Walking All FilesIterate over files.forfileinfs.files():# do something# Or using the class' iter method...forfileinfs:# do somethingIterate over folders that contain files (i.e. ignore the nested subfolders that only contain folders).forfolderinfs.folders():# do somethingComputing SizeCompute the size in bytes of all files in therootdirectory.total_bytes=fs.size()Count the total number of files.total_files=fs.count()# Or via len()...total_files=len(fs)For more details, please see the full documentation athttp://hashfs.readthedocs.org.Changelogv0.7.2 (2019-10-24)Fix out-of-memory issue when computing file ID hashes of large files.v0.7.1 (2018-10-13)Replace usage ofdistutils.dir_util.mkpathwithos.path.makedirs.v0.7.0 (2016-04-19)Useshutil.moveinstead ofshutil.copyto move temporary file created duringputoperation toHashFSdirectory.v0.6.0 (2015-10-19)Add fasterscandirpackage for iterating over files/folders when platform is Python < 3.5. Scandir implementation was added toosmodule starting with Python 3.5.v0.5.0 (2015-07-02)Rename private methodHashFS.copytoHashFS._copy.Addis_duplicateattribute toHashAddress.MakeHashFS.put()returnHashAddresswithis_duplicate=Truewhen file with same hash already exists on disk.v0.4.0 (2015-06-03)AddHashFS.size()method that returns the size of all files in bytes.AddHashFS.count()/HashFS.__len__()methods that return the count of all files.AddHashFS.__iter__()method to support iteration. Proxies toHashFS.files().AddHashFS.__contains__()method to supportinoperator. Proxies toHashFS.exists().Don’t create the root directory (if it doesn’t exist) until at least one file has been added.FixHashFS.repair()not usingextensionsargument properly.v0.3.0 (2015-06-02)RenameHashFS.lengthparameter/property towidth. (breaking change)v0.2.0 (2015-05-29)RenameHashFS.gettoHashFS.open. (breaking change)AddHashFS.get()method that returns aHashAddressorNonegiven a file ID or path.v0.1.0 (2015-05-28)AddHashFS.get()method that retrieves a reader object given a file ID or path.AddHashFS.delete()method that deletes a file ID or path.AddHashFS.folders()method that returns the folder paths that directly contain files (i.e. subpaths that only contain folders are ignored).AddHashFS.detokenize()method that returns the file ID contained in a file path.AddHashFS.repair()method that reindexes any files under root directory whose file path doesn’t not match its tokenized file ID.RenameAddressclasss toHashAddress. (breaking change)RenameHashAddress.digesttoHashAddress.id. (breaking change)RenameHashAddress.pathtoHashAddress.abspath. (breaking change)AddHashAddress.relpathwhich represents path relative toHashFS.root.v0.0.1 (2015-05-27)First release.AddHashFSclass.AddHashFS.put()method that saves a file path or file-like object by content hash.AddHashFS.files()method that returns all files under root directory.AddHashFS.exists()which checks either a file hash or file path for existence.