Package: httpx-s3-client
httpx-s3-clientThe simple module for putting and getting object from Amazon S3 compatible endpoints.Installationpipinstallhttpx-s3-clientUsagefromhttpimportHTTPStatusfromhttpximportAsyncClientfromhttpx_s3_clientimportS3Clientclient=S3Client(url="http://s3-url",client=AsyncClient(),access_key_id="key-id",secret_access_key="hackme",region="us-east-1")# Upload str object to bucket "bucket" and key "str"resp=awaitclient.put("bucket/str","hello, world")assertresp.status_code==HTTPStatus.OK# Upload bytes object to bucket "bucket" and key "bytes"resp=awaitclient.put("bucket/bytes",b"hello, world")assertresp.status_code==HTTPStatus.OK# Upload AsyncIterable to bucket "bucket" and key "iterable"asyncdefgen():yieldb'some bytes'resp=awaitclient.put("bucket/file",gen())assertresp.status_code==HTTPStatus.OK# Upload file to bucket "bucket" and key "file"resp=awaitclient.put_file("bucket/file","/path_to_file")assertresp.status_code==HTTPStatus.OK# Check object exists using bucket+keyresp=awaitclient.head("bucket/key")assertresp.status_code==HTTPStatus.OK# Get object by bucket+keyresp=awaitclient.get("bucket/key")data=resp.content# Delete object using bucket+keyresp=awaitclient.delete("bucket/key")assertresp==HTTPStatus.NO_CONTENT# List objects by prefixasyncforresultinclient.list_objects_v2("bucket/",prefix="prefix"):# Each result is a list of metadata objects representing an object# stored in the bucket.do_work(result)Bucket may be specified as subdomain or in object name:importhttpxfromhttpx_s3_clientimportS3Clientclient=S3Client(url="http://bucket.your-s3-host",client=httpx.AsyncClient())resp=awaitclient.put("key",gen())...client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient())resp=awaitclient.put("bucket/key",gen())...client=S3Client(url="http://your-s3-host/bucket",client=httpx.AsyncClient())resp=awaitclient.put("key",gen())...Auth may be specified with keywords or in URL:importhttpxfromhttpx_s3_clientimportS3Clientclient_credentials_as_kw=S3Client(url="http://your-s3-host",access_key_id="key_id",secret_access_key="access_key",client=httpx.AsyncClient(),)client_credentials_in_url=S3Client(url="http://key_id:access_key@your-s3-host",client=httpx.AsyncClient(),)CredentialsBy defaultS3Clienttrying to collect all available credentials from keyword
arguments likeaccess_key_id=andsecret_access_key=, after that from the
username and password from passedurlargument, so the nex step is environment
variables parsing and the last source for collection is the config file.You can pass credentials explicitly usinghttpx_s3_client.credentialsmodule.httpx_s3_client.credentials.StaticCredentialsimporthttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportStaticCredentialscredentials=StaticCredentials(access_key_id='aaaa',secret_access_key='bbbb',region='us-east-1',)client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)httpx_s3_client.credentials.URLCredentialsimporthttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportURLCredentialsurl="http://key@hack-me:your-s3-host"credentials=URLCredentials(url,region="us-east-1")client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)httpx_s3_client.credentials.EnvironmentCredentialsimporthttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportEnvironmentCredentialscredentials=EnvironmentCredentials(region="us-east-1")client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)httpx_s3_client.credentials.ConfigCredentialsUsing user config file:importhttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportConfigCredentialscredentials=ConfigCredentials()# Will be used ~/.aws/credentials configclient=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)Using the custom config location:importhttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportConfigCredentialscredentials=ConfigCredentials("~/.my-custom-aws-credentials")client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)httpx_s3_client.credentials.merge_credentialsThis function collect all passed credentials instances and return a new one
which contains all non-blank fields from passed instances. The first argument
has more priority.importhttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimport(ConfigCredentials,EnvironmentCredentials,merge_credentials)credentials=merge_credentials(EnvironmentCredentials(),ConfigCredentials(),)client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),credentials=credentials,)httpx_s3_client.credentials.MetadataCredentialsTrying to get credentials from the metadata service:importhttpxfromhttpx_s3_clientimportS3Clientfromhttpx_s3_client.credentialsimportMetadataCredentialscredentials=MetadataCredentials()# start refresh credentials from metadata serverawaitcredentials.start()client=S3Client(url="http://your-s3-host",client=httpx.AsyncClient(),)awaitcredentials.stop()Multipart uploadFor uploading large filesmultipart uploadingcan be used. It allows you to asynchronously upload multiple parts of a file
to S3.
S3Client handles retries of part uploads and calculates part hash for integrity checks.importhttpxfromhttpx_s3_clientimportS3Clientclient=S3Client(url="http://your-s3-host",client=httpx.AsyncClient())awaitclient.put_file_multipart("test/bigfile.csv",headers={"Content-Type":"text/csv",},workers_count=8,)Parallel download to fileS3 supportsGETrequests withRangeheader. It's possible to download
objects in parallel with multiple connections for speedup.
S3Client handles retries of partial requests and makes sure that file won't
be changed during download withETagheader.
If your system supportspwritesyscall (Linux, macOS, etc.) it will be used to
write simultaneously to a single file. Otherwise, each worker will have own file
which will be concatenated after downloading.importhttpxfromhttpx_s3_clientimportS3Clientclient=S3Client(url="http://your-s3-host",client=httpx.AsyncClient())awaitclient.get_file_parallel("dump/bigfile.csv","/home/user/bigfile.csv",workers_count=8,)