Package: smart-open
What?smart_openis a Python 3 library forefficient streaming of very large filesfrom/to storages such as S3, GCS, Azure Blob Storage, HDFS, WebHDFS, HTTP, HTTPS, SFTP, or local filesystem. It supports transparent, on-the-fly (de-)compression for a variety of different formats.smart_openis a drop-in replacement for Python’s built-inopen(): it can do anythingopencan (100% compatible, falls back to nativeopenwherever possible), plus lots of nifty extra stuff on top.Python 2.7 is no longer supported. If you need Python 2.7, please usesmart_open 1.10.1,the last version to support Python 2.Why?Working with large remote files, for example using Amazon’sboto3Python library, is a pain.boto3’sObject.upload_fileobj()andObject.download_fileobj()methods require gotcha-prone boilerplate to use successfully, such as constructing file-like object wrappers.smart_openshields you from that. It builds on boto3 and other remote storage libraries, but offers aclean unified Pythonic API. The result is less code for you to write and fewer bugs to make.How?smart_openis well-tested, well-documented, and has a simple Pythonic API:>>>fromsmart_openimportopen>>>>>># stream lines from an S3 object>>>forlineinopen('s3://commoncrawl/robots.txt'):...print(repr(line))...break'User-Agent: *\n'>>># stream from/to compressed files, with transparent (de)compression:>>>forlineinopen('smart_open/tests/test_data/1984.txt.gz',encoding='utf-8'):...print(repr(line))'It was a bright cold day in April, and the clocks were striking thirteen.\n''Winston Smith, his chin nuzzled into his breast in an effort to escape the vile\n''wind, slipped quickly through the glass doors of Victory Mansions, though not\n''quickly enough to prevent a swirl of gritty dust from entering along with him.\n'>>># can use context managers too:>>>withopen('smart_open/tests/test_data/1984.txt.gz')asfin:...withopen('smart_open/tests/test_data/1984.txt.bz2','w')asfout:...forlineinfin:...fout.write(line)74807879>>># can use any IOBase operations, like seek>>>withopen('s3://commoncrawl/robots.txt','rb')asfin:...forlineinfin:...print(repr(line.decode('utf-8')))...break...offset=fin.seek(0)# seek to the beginning...print(fin.read(4))'User-Agent: *\n'b'User'>>># stream from HTTP>>>forlineinopen('http://example.com/index.html'):...print(repr(line))...break'<!doctype html>\n'Other examples of URLs thatsmart_openaccepts:s3://my_bucket/my_key
s3://my_key:my_secret@my_bucket/my_key
s3://my_key:my_secret@my_server:my_port@my_bucket/my_key
gs://my_bucket/my_blob
azure://my_bucket/my_blob
hdfs:///path/file
hdfs://path/file
webhdfs://host:port/path/file
./local/path/file
~/local/path/file
local/path/file
./local/path/file.gz
file:///home/user/file
file:///home/user/file.bz2
[ssh|scp|sftp]://username@host//path/file
[ssh|scp|sftp]://username@host/path/file
[ssh|scp|sftp]://username:password@host/path/fileDocumentationInstallationsmart_opensupports a wide range of storage solutions, including AWS S3, Google Cloud and Azure.
Each individual solution has its own dependencies.
By default,smart_opendoes not install any dependencies, in order to keep the installation size small.
You can install these dependencies explicitly using:pip install smart_open[azure] # Install Azure deps
pip install smart_open[gcs] # Install GCS deps
pip install smart_open[s3] # Install S3 depsOr, if you don’t mind installing a large number of third party libraries, you can install all dependencies using:pip install smart_open[all]Be warned that this option increases the installation size significantly, e.g. over 100MB.If you’re upgrading fromsmart_openversions 2.x and below, please check out theMigration Guide.Built-in helpFor detailed API info, see the online help:help('smart_open')or clickhereto view the help in your browser.More examplesFor the sake of simplicity, the examples below assume you have all the dependencies installed, i.e. you have done:pip install smart_open[all]>>>importos,boto3>>>fromsmart_openimportopen>>>>>># stream content *into* S3 (write mode) using a custom session>>>session=boto3.Session(...aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],...aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],...)>>>url='s3://smart-open-py37-benchmark-results/test.txt'>>>withopen(url,'wb',transport_params={'client':session.client('s3')})asfout:...bytes_written=fout.write(b'hello world!')...print(bytes_written)12# stream from HDFSforlineinopen('hdfs://user/hadoop/my_file.txt',encoding='utf8'):print(line)# stream from WebHDFSforlineinopen('webhdfs://host:port/user/hadoop/my_file.txt'):print(line)# stream content *into* HDFS (write mode):withopen('hdfs://host:port/user/hadoop/my_file.txt','wb')asfout:fout.write(b'hello world')# stream content *into* WebHDFS (write mode):withopen('webhdfs://host:port/user/hadoop/my_file.txt','wb')asfout:fout.write(b'hello world')# stream from a completely custom s3 server, like s3proxy:forlineinopen('s3u://user:secret@host:port@mybucket/mykey.txt'):print(line)# Stream to Digital Ocean Spaces bucket providing credentials from boto3 profilesession=boto3.Session(profile_name='digitalocean')client=session.client('s3',endpoint_url='https://ams3.digitaloceanspaces.com')transport_params={'client':client}withopen('s3://bucket/key.txt','wb',transport_params=transport_params)asfout:fout.write(b'here we stand')# stream from GCSforlineinopen('gs://my_bucket/my_file.txt'):print(line)# stream content *into* GCS (write mode):withopen('gs://my_bucket/my_file.txt','wb')asfout:fout.write(b'hello world')# stream from Azure Blob Storageconnect_str=os.environ['AZURE_STORAGE_CONNECTION_STRING']transport_params={'client':azure.storage.blob.BlobServiceClient.from_connection_string(connect_str),}forlineinopen('azure://mycontainer/myfile.txt',transport_params=transport_params):print(line)# stream content *into* Azure Blob Storage (write mode):connect_str=os.environ['AZURE_STORAGE_CONNECTION_STRING']transport_params={'client':azure.storage.blob.BlobServiceClient.from_connection_string(connect_str),}withopen('azure://mycontainer/my_file.txt','wb',transport_params=transport_params)asfout:fout.write(b'hello world')Compression HandlingThe top-levelcompressionparameter controls compression/decompression behavior when reading and writing.
The supported values for this parameter are:infer_from_extension(default behavior)disable.gz.bz2By default,smart_opendetermines the compression algorithm to use based on the file extension.>>>fromsmart_openimportopen,register_compressor>>>withopen('smart_open/tests/test_data/1984.txt.gz')asfin:...print(fin.read(32))ItwasabrightcolddayinApriYou can override this behavior to either disable compression, or explicitly specify the algorithm to use.
To disable compression:>>>fromsmart_openimportopen,register_compressor>>>withopen('smart_open/tests/test_data/1984.txt.gz','rb',compression='disable')asfin:...print(fin.read(32))b'\x1f\x8b\x08\x08\x85F\x94\\\x00\x031984.txt\x005\x8f=r\xc3@\x08\x85{\x9d\xe2\x1d@'To specify the algorithm explicitly (e.g. for non-standard file extensions):>>>fromsmart_openimportopen,register_compressor>>>withopen('smart_open/tests/test_data/1984.txt.gzip',compression='.gz')asfin:...print(fin.read(32))ItwasabrightcolddayinApriYou can also easily add support for other file extensions and compression formats.
For example, to open xz-compressed files:>>>importlzma,os>>>fromsmart_openimportopen,register_compressor>>>def_handle_xz(file_obj,mode):...returnlzma.LZMAFile(filename=file_obj,mode=mode,format=lzma.FORMAT_XZ)>>>register_compressor('.xz',_handle_xz)>>>withopen('smart_open/tests/test_data/1984.txt.xz')asfin:...print(fin.read(32))ItwasabrightcolddayinAprilzmais in the standard library in Python 3.3 and greater.
For 2.7, usebackports.lzma.Transport-specific Optionssmart_opensupports a wide range of transport options out of the box, including:S3HTTP, HTTPS (read-only)SSH, SCP and SFTPWebHDFSGCSAzure Blob StorageEach option involves setting up its own set of parameters.
For example, for accessing S3, you often need to set up authentication, like API keys or a profile name.smart_open’sopenfunction accepts a keyword argumenttransport_paramswhich accepts additional parameters for the transport layer.
Here are some examples of using this parameter:>>>importboto3>>>fin=open('s3://commoncrawl/robots.txt',transport_params=dict(client=boto3.client('s3')))>>>fin=open('s3://commoncrawl/robots.txt',transport_params=dict(buffer_size=1024))For the full list of keyword arguments supported by each transport option, see the documentation:help('smart_open.open')S3 Credentialssmart_openuses theboto3library to talk to S3.boto3has severalmechanismsfor determining the credentials to use.
By default,smart_openwill defer toboto3and let the latter take care of the credentials.
There are several ways to override this behavior.The first is to pass aboto3.Clientobject as a transport parameter to theopenfunction.
You can customize the credentials when constructing the session for the client.smart_openwill then use the session when talking to S3.session=boto3.Session(aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY,aws_session_token=SESSION_TOKEN,)client=session.client('s3',endpoint_url=...,config=...)fin=open('s3://bucket/key',transport_params={'client':client})Your second option is to specify the credentials within the S3 URL itself:fin=open('s3://aws_access_key_id:aws_secret_access_key@bucket/key',...)Important: The two methods above aremutually exclusive. If you pass an AWS clientandthe URL contains credentials,smart_openwill ignore the latter.Important:smart_openignores configuration files from the olderbotolibrary.
Port your oldbotosettings toboto3in order to use them withsmart_open.S3 Advanced UsageAdditional keyword arguments can be propagated to the boto3 methods that are used bysmart_openunder the hood using theclient_kwargstransport parameter.For instance, to upload a blob with Metadata, ACL, StorageClass, these keyword arguments can be passed tocreate_multipart_upload(docs).kwargs={'Metadata':{'version':2},'ACL':'authenticated-read','StorageClass':'STANDARD_IA'}fout=open('s3://bucket/key','wb',transport_params={'client_kwargs':{'S3.Client.create_multipart_upload':kwargs}})Iterating Over an S3 Bucket’s ContentsSince going over all (or select) keys in an S3 bucket is a very common operation, there’s also an extra functionsmart_open.s3.iter_bucket()that does this efficiently,processing the bucket keys in parallel(using multiprocessing):>>>fromsmart_openimports3>>># we use workers=1 for reproducibility; you should use as many workers as you have cores>>>bucket='silo-open-data'>>>prefix='Official/annual/monthly_rain/'>>>forkey,contentins3.iter_bucket(bucket,prefix=prefix,accept_key=lambdakey:'/201'inkey,workers=1,key_limit=3):...print(key,round(len(content)/2**20))Official/annual/monthly_rain/2010.monthly_rain.nc13Official/annual/monthly_rain/2011.monthly_rain.nc13Official/annual/monthly_rain/2012.monthly_rain.nc13GCS Credentialssmart_openuses thegoogle-cloud-storagelibrary to talk to GCS.google-cloud-storageuses thegoogle-cloudpackage under the hood to handle authentication.
There are severaloptionsto provide
credentials.
By default,smart_openwill defer togoogle-cloud-storageand let it take care of the credentials.To override this behavior, pass agoogle.cloud.storage.Clientobject as a transport parameter to theopenfunction.
You cancustomize the credentialswhen constructing the client.smart_openwill then use the client when talking to GCS. To follow allow with
the example below,refer to Google’s guideto setting up GCS authentication with a service account.importosfromgoogle.cloud.storageimportClientservice_account_path=os.environ['GOOGLE_APPLICATION_CREDENTIALS']client=Client.from_service_account_json(service_account_path)fin=open('gs://gcp-public-data-landsat/index.csv.gz',transport_params=dict(client=client))If you need more credential options, you can create an explicitgoogle.auth.credentials.Credentialsobject
and pass it to the Client. To create an API token for use in the example below, refer to theGCS authentication guide.importosfromgoogle.auth.credentialsimportCredentialsfromgoogle.cloud.storageimportClienttoken=os.environ['GOOGLE_API_TOKEN']credentials=Credentials(token=token)client=Client(credentials=credentials)fin=open('gs://gcp-public-data-landsat/index.csv.gz',transport_params={'client':client})GCS Advanced UsageAdditional keyword arguments can be propagated to the GCS open method (docs), which is used bysmart_openunder the hood, using theblob_open_kwargstransport parameter.Additional blob properties (docs) can be set before an upload, as long as they are not read-only, using theblob_propertiestransport parameter.open_kwargs={'predefined_acl':'authenticated-read'}properties={'metadata':{'version':2},'storage_class':'COLDLINE'}fout=open('gs://bucket/key','wb',transport_params={'blob_open_kwargs':open_kwargs,'blob_properties':properties})Azure Credentialssmart_openuses theazure-storage-bloblibrary to talk to Azure Blob Storage.
By default,smart_openwill defer toazure-storage-bloband let it take care of the credentials.Azure Blob Storage does not have any ways of inferring credentials therefore, passing aazure.storage.blob.BlobServiceClientobject as a transport parameter to theopenfunction is required.
You cancustomize the credentialswhen constructing the client.smart_openwill then use the client when talking to. To follow allow with
the example below,refer to Azure’s guideto setting up authentication.importosfromazure.storage.blobimportBlobServiceClientazure_storage_connection_string=os.environ['AZURE_STORAGE_CONNECTION_STRING']client=BlobServiceClient.from_connection_string(azure_storage_connection_string)fin=open('azure://my_container/my_blob.txt',transport_params={'client':client})If you need more credential options, refer to theAzure Storage authentication guide.Azure Advanced UsageAdditional keyword arguments can be propagated to thecommit_block_listmethod (docs), which is used bysmart_openunder the hood for uploads, using theblob_kwargstransport parameter.kwargs={'metadata':{'version':2}}fout=open('azure://container/key','wb',transport_params={'blob_kwargs':kwargs})Drop-in replacement ofpathlib.Path.opensmart_open.opencan also be used withPathobjects.
The built-inPath.open()is not able to read text from compressed files, so usepatch_pathlibto replace it withsmart_open.open()instead.
This can be helpful when e.g. working with compressed files.>>>frompathlibimportPath>>>fromsmart_open.smart_open_libimportpatch_pathlib>>>>>>_=patch_pathlib()# replace `Path.open` with `smart_open.open`>>>>>>path=Path("smart_open/tests/test_data/crime-and-punishment.txt.gz")>>>>>>withpath.open("r")asinfile:...print(infile.readline()[:41])Вначалеиюля,вчрезвычайножаркоевремяHow do I …?Seethis document.Extendingsmart_openSeethis document.Testingsmart_opensmart_opencomes with a comprehensive suite of unit tests.
Before you can run the test suite, install the test dependencies:pip install -e .[test]Now, you can run the unit tests:pytest smart_openThe tests are also run automatically withTravis CIon every commit push & pull request.Comments, bug reportssmart_openlives onGithub. You can file
issues or pull requests there. Suggestions, pull requests and improvements welcome!smart_openis open source software released under theMIT license.
Copyright (c) 2015-nowRadim Řehůřek.