Package: omnio
omnioPython library for opening URIs as streaming file-like objectsThis library provides theomnio.open()function with an interface very
similar to that of the built in Pythonopenfunction.  The difference is
that while Python'sopenoperates only on local filesystem paths,omnioaccepts URIs as well.It currently supportsfile,http, ands3URIs, though it may be
expanded to support additional schemes in the future.In addition, it supports compression and decompression of streams with gzip
or bz2.FeaturesAPI is a superset of Python 3's built-in open() functionBased on Python 3iomoduleGzip and bzip2 support both for reading and writingLocal file support using standard libraryHTTP support usingrequestsS3 support usingboto3ExamplesRead a local binary file:>>> with omnio.open('example.bin', 'r') as f:
>>>     data = f.read()Read a local text file:>>> with omnio.open('example.txt', 'rt') as f:
>>>     text = f.read()Read a text file from the web:>>> with omnio.open('https://example.com/example.txt', 'rt') as f:
>>>     text = f.read()Read a gzipped text file from the web, uncompressing on the fly:>>> with omnio.open('https://example.com/example.txt.gz', 'rtz') as f:
>>>     text = f.read()Read a text file from s3:>>> with omnio.open('s3://my-bucket/my-key', 'rt') as f:
>>>     text = f.read()Write a megabyte of random data to s3, compressing with bzip2:>>> import os
>>> data = os.urandom(1024**2)
>>> with omnio.open('s3://my-bucket/my-key', 'wbj') as f:
>>>     f.write(data)Read a bzip2 compressed csv file into a list of data rows:>>> import csv
>>> with omnio.open('data/example_data.csv.bz2', 'rtj') as f:
>>>     reader = csv.reader(f)
>>>     data = list(reader)APIomnio.open()The top-level API provides of a single function intended to be
referenced asomnio.open(). This function API is designed to mimic
Python 3's built-inopen()as much as possible, and should normally
be able to be used as a drop-in replacement.Signature:omnio.open(uri, mode='rb', encoding=None, errors=None, newline=None)Returns:A file-like object whose type depends on the scheme and the mode.Parameters:uri-- URI or local path. Supported URI schemes arefile,http, ands3. Local paths may be specified by as ordinary path
strings.mode-- Optional string that specifies the mode in which the
file is opened. It defaults to 'rb' which means open for reading
in binary mode. Supported modes are documented below.Modes:CharacterMeaning'r'open for reading (default)'w'open for writing, truncating the file first'b'binary mode (default)'t'text mode'z'use gzip compression'j'use bzip2 compressionThese characters can be composed into valid modes. Binary mode is
always the default, so some mode specifications are equivalent.
The complete list of supported modes are as follows:ModeMeaning'r', 'rb'read bytes'rt'read and decode to unicode'rz', 'rbz'read, decompressing gzip to bytes'rj', 'rbj'read, decompressing bzip2 to bytes'rtz'read, decompress gzip to bytes, decode to unicode'rtj'read, decompress bzip2 to bytes, decode to unicode'w', 'wb'write bytes'wt'write unicode, encoding to bytes'wz', 'wbz'write bytes, compress with gzip'wj', 'wbj'write bytes, compress with bzip2'wtz'write unicode, encode to bytes, compress with gzip'wtj'write unicode, encode to bytes, compress with bzip2Some keyword arguments may be applicable to only certain modes. For
example,encodingonly applies to 't' (text) modes.Some schemes may not support some modes.  For example, the http
scheme currently does not support any 'w' (write) modes.omnio.globTheglobsubmodule is intended to be a drop-in replacement for the
standard libraryglobmodule. It provides the functionsescape(),glob(), andiglob()with the signature of each matching that of
the original.The difference is thatglob()andiglob()optionally accept URIs.
Currently, local paths,file://, ands3://URIs are supported.ConfigurationTheomnio.openfunction accepts an optionalconfigparameter. This
allows for specifying scheme-specific configuration.Thedefault_config()method returns a config dictionary with all supported
keys defined along with their default values.>>> import omnio
>>> omnio.default_config()
{'file': {}, 'http': {'content_iter_chunk_size': 512}, 's3': {'upload_part_size': 5242880, 'boto_client_config_args': [], 'boto_client_config_kwargs': {}}}To specify alternate values for these parameters, instantiate a default
config, update the dict with the desired values and pass it as a keyword arg
to theomnio.open()function.>>> config = omnio.default_config()
>>> config["s3"]["boto_client_config_kwargs"] = {"read_timeout": 600}
>>> with omnio.open("s3://my-bucket/my-key", "rt", config=config) as fd:
    fd.read()