Package: megfile
megfile - Megvii FILE libraryDocs:http://megvii-research.github.io/megfilemegfileprovides a silky operation experience with different backends (currently including local file system and s3), which enable you to focus more on the logic of your own project instead of the question of "Which backend is used for this file?"megfileprovides:Almost unified file system operation experience. Target path can be easily moved from local file system to s3.Complete boundary case handling. Even the most difficult (or even you can't even think of) boundary conditions,megfilecan help you easily handle it.Perfect type hints and built-in documentation. You can enjoy the IDE's auto-completion and static checking.Semantic version and upgrade guide, which allows you enjoy the latest features easily.megfile's advantages are:smart_opencan open resources that use various protocols. Especially, reader / writer of s3 inmegfileis implemented with multi-thread, which is faster than known competitors.smart_globis available on majority protocols. And it supports zsh extended pattern syntax of[], e.g.s3://bucket/video.{mp4,avi}.All-inclusive functions likesmart_exists/smart_stat/smart_sync. If you don't find the functions you want,submit an issue.Compatible withpathlib.Pathinterface, referring toSmartPathand other protocol classes likeS3Path.Support Protocolsfs(local filesystem)s3sftphttpstdiohdfs:pip install 'megfile[hdfs]'Quick StartPath string inmegfilealmost isprotocol://path/to/file, for examples3://bucketA/key. But sftp path is a little different, format issftp://[username[:password]@]hostname[:port]//file_path, and relative path is replace//file_pathto/file_path.
Here's an example of writing a file to s3 / fs, syncing to local, reading and finally deleting it.Functional Interfacefrommegfileimportsmart_open,smart_exists,smart_sync,smart_remove,smart_glob# open a file in s3 bucketwithsmart_open('s3://playground/megfile-test','w')asfp:fp.write('megfile is not silver bullet')# test if file in s3 bucket existsmart_exists('s3://playground/megfile-test')# or in local file systemsmart_exists('/tmp/playground/megfile-test')# copy files or directoriessmart_sync('s3://playground/megfile-test','/tmp/playground/megfile-test')# remove files or directoriessmart_remove('s3://playground/megfile-test')# glob files or directories in s3 bucketsmart_glob('s3://playground/megfile-?.{mp4,avi}')SmartPath InterfaceSmartPathhas a similar interface with pathlib.Path.frommegfile.smart_pathimportSmartPathpath=SmartPath('s3://playground/megfile-test')ifpath.exists():withpath.open()asf:result=f.read(7)assertresult==b'megfile'Command Line Interface$megfile--help# see what you can do$megfilelss3://playground/
$megfilels-l-hs3://playground/

$megfilecats3://playground/megfile-test

$megfilecps3://playground/megfile-test/tmp/playground/megfile-testInstallationPyPIpip3installmegfileYou can specify megfile version as wellpip3install"megfile~=0.0"Build from Sourcemegfile can be installed from sourcegitclonegit@github.com:megvii-research/megfile.gitcdmegfile
pip3install-U.Development Environmentgitclonegit@github.com:megvii-research/megfile.gitcdmegfile
pip3install-rrequirements.txt-rrequirements-dev.txtConfigurationUsings3as an example, the following describes the configuration methods. For more details, please refer toConfiguration.You can use environments and configuration file for configuration, and priority is that environment variables take precedence over configuration file.Use environmentsYou can use environments to setup authentication credentials for your s3 account:AWS_ACCESS_KEY_ID: access keyAWS_SECRET_ACCESS_KEY: secret keyOSS_ENDPOINT: endpoint url of s3AWS_S3_ADDRESSING_STYLE: addressing styleUse commandYou can update config file withmegfilecommand easyly:megfile config s3 [OPTIONS] AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY$ megfile config s3 accesskey secretkey

# for aliyun
$ megfile config s3 accesskey secretkey \
--addressing-style virtual \
--endpoint-url http://oss-cn-hangzhou.aliyuncs.com \You can get the configuration from~/.aws/credentials, like:[default]
aws_secret_access_key = accesskey
aws_access_key_id = secretkey

s3 =
    addressing_style = virtual
    endpoint_url = http://oss-cn-hangzhou.aliyuncs.comHow to ContributeWe welcome everyone to contribute code to themegfileproject, but the contributed code needs to meet the following conditions as much as possible:You can submit code even if the code doesn't meet conditions. The project members will evaluate and assist you in making code changesCode format: Your code needs to passcode format check.megfileusesyapfas lint tool and the version is locked at 0.27.0. The version lock may be removed in the futureStatic check: Your code needs completetype hint.megfileusespytypeas static check tool. Ifpytypefailed in static check, use# pytype: disable=XXXto disable the error and please tell us why you disable it.Note: Becausepytypedoesn't support variable type annation, the variable type hint format introduced by py36 cannot be used.i.e.variable: intis invalid, replace it withvariable  # type: intTest: Your code needs completeunit testcoverage.megfileusespyfakefsandmotoas local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctnessYou can help to improvemegfilein many ways:Write code.Improvedocumentation.Report or investigatebugs and issues.If you find any problem or have any improving suggestion,submit a new issuseas well. We will reply as soon as possible and evaluate whether to adopt.Reviewpull requests.Starmegfilerepo.Recommendmegfileto your friends.Any other form of contribution is welcomed.