Package: hfutils
hfutilsUseful utilities for huggingfaceQuick StartTo get started withhfutils, you can install it using PyPI:pipinstallhfutilsAlternatively, you can install it from the source code:gitclonehttps://github.com/deepghs/hfutils.gitcdhfutils
pipinstall.Verify the installation by checking the version:hfutils-vIf Python is not available in your local environment, we recommend downloading the pre-compiled executable version from
thereleases.Upload DataUpload data to repositories using the following commands:# Upload a single file to the repositoryhfutilsupload-ryour/repository-i/your/local/file-ffile/in/your/repo# Upload files in a directory as an archive file to the repository# More formats of archive files are supported# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.htmlhfutilsupload-ryour/repository-i/your/local/directory-aarchive/file/in/your/repo.zip# Upload files in a directory as a directory tree to the repositoryhfutilsupload-ryour/repository-i/your/local/directory-ddir/in/your/repoYou can achieve the same using the Python API:fromhfutils.operateimportupload_file_to_file,upload_directory_as_archive,upload_directory_as_directory# Upload a single file to the repositoryupload_file_to_file(local_file='/your/local/file',repo_id='your/repository',file_in_repo='file/in/your/repo')# Upload files in a directory as an archive file to the repository# More formats of archive files are supported# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.htmlupload_directory_as_archive(local_directory='/your/local/directory',repo_id='your/repository',archive_in_repo='archive/file/in/your/repo.zip')# Upload files in a directory as a directory tree to the repositoryupload_directory_as_directory(local_directory='/your/local/directory',repo_id='your/repository',path_in_repo='dir/in/your/repo')Explore additional options for uploading:hfutilsupload-hDownload DataDownload data from repositories using the following commands:# Download a single file from the repositoryhfutilsdownload-ryour/repository-o/your/local/file-ffile/in/your/repo# Download an archive file from the repository and extract it to the given directory# More formats of archive files are supported# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.htmlhfutilsdownload-ryour/repository-o/your/local/directory-aarchive/file/in/your/repo.zip# Download files from the repository as a directory treehfutilsdownload-ryour/repository-o/your/local/directory-ddir/in/your/repoUse the Python API for the same functionality:fromhfutils.operateimportdownload_file_to_file,download_archive_as_directory,download_directory_as_directory# Download a single file from the repositorydownload_file_to_file(local_file='/your/local/file',repo_id='your/repository',file_in_repo='file/in/your/repo')# Download an archive file from the repository and extract it to the given directory# More formats of archive files are supported# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.htmldownload_archive_as_directory(local_directory='/your/local/directory',repo_id='your/repository',file_in_repo='archive/file/in/your/repo.zip',)# Download files from the repository as a directory treedownload_directory_as_directory(local_directory='/your/local/directory',repo_id='your/repository',dir_in_repo='dir/in/your/repo')Explore additional options for downloading:hfutilsdownload-hSupported FormatsBy default, we support thezipandtarformats, including.zip,.tar,.tar.gz,.tar.bz2, and.tar.xz.If you require support forrarand7zfiles, install the extra dependencies using the following command:pipinstallhfutils[rar,7z]NOTE:Creating RAR archive files is not supported. We use therarfilelibrary,
which lacks the functionality for creating RAR files.How to Access Private RepositoriesSimply configure theHF_TOKENenvironment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.How to Use Hf-Transfer for AccelerationIf you are using the PyPI CLI, you need to installhfutilswith the following command:pipinstallhfutils[transfer]If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.Enable Hf-Transfer acceleration by setting the environment variableHF_HUB_ENABLE_HF_TRANSFERto1.How to Change the Temporary DirectoryThe temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk
space.If your disk, especially the C drive on Windows, does not have sufficient space, simply set theTMPDIRto designate
another directory as the temporary directory.