Package: cs.buffer
Facilities to do with buffers, particularly CornuCopyBuffer,
an automatically refilling buffer to support parsing of data streams.Latest release 20240201:
CornuCopyBuffer: read1() method shorthand for read(..,one_fetch=True).ClassCopyingIteratorWrapper for an iterator that copies every item retrieved to a callable.MethodCopyingIterator.__init__(self, it, copy_to):
Initialise with the iteratoritand the callablecopy_to.ClassCornuCopyBuffer(cs.deco.Promotable)An automatically refilling buffer intended to support parsing
of data streams.Its purpose is to aid binary parsers
which do not themselves need to handle sources specially;CornuCopyBuffers are trivially made frombytes,
iterables ofbytesand file-like objects.
Seecs.binaryfor convenient parsing classes
which work againstCornuCopyBuffers.Attributes:buf: the first of any buffered leading chunks
buffer of unparsed data from the input, available
for direct inspection by parsers;
normally however parsers will use.extendand.take.offset: the logical offset of the buffer; this excludes
buffered data and unconsumed input dataNote: the initialiser may supply a cleanup function;
although this will be called via the buffer's.__del__method
a prudent user of a buffer should call the.close()method
when finished with the buffer to ensure prompt cleanup.The primary methods supporting parsing of data streams are.extend()andtake().
Calling.extend(min_size)arranges that the internal buffer
contains at leastmin_sizebytes.
Calling.take(size)fetches exactlysizebytes from the
internal buffer and the input source if necessary and returns
them, adjusting the internal buffer.len(CornuCopyBuffer) returns the length of any buffered data.bool(CornuCopyBuffer) tests whether len() > 0.Indexing aCornuCopyBufferaccesses the buffered data only,
returning an individual byte's value (anint).ACornuCopyBufferis also iterable, yielding data in whatever
sizes come from itsinput_datasource, preceeded by any
content in the internal buffer.ACornuCopyBufferalso supports the file methods.read,.telland.seeksupporting drop in use of the buffer in
many file contexts. Backward seeks are not supported..seekwill take advantage of theinput_data's .seek method if it
has one, otherwise it will use consume theinput_dataas required.MethodCornuCopyBuffer.__init__(self, input_data, buf=None, offset=0, seekable=None, copy_offsets=None, copy_chunks=None, close=None, progress=None):
Prepare the buffer.Parameters:input_data: an iterable of data chunks (bytes-like instances);
if your data source is a file see the.from_filefactory;
if your data source is a file descriptor see the.from_fdfactory.buf: if notNone, the initial state of the parse bufferoffset: logical offset of the start of the buffer, default0seekable: whetherinput_datahas a working.seekmethod;
the default isNonemeaning that it will be attempted on
the first skip or seekcopy_offsets: if notNone, a callable for parsers to
report pertinent offsets via the buffer's.report_offsetmethodcopy_chunks: if notNone, every fetched data chunk is
copied to this callableTheinput_datais an iterable whose iterator may have
some optional additional properties:seek: if present, this is a seek method after the fashion
offile.seek; the buffer'sseek,skipandskiptomethods will take advantage of this if available.offset: the current byte offset of the iterator; this
is used during the buffer initialisation to computeinput_data_displacement, the difference between the
buffer's logical offset and the input data iterable's logical offset;
if unavailable during initialisation this is presumed to
be0.end_offset: the end offset of the iterator if known.close: an optional callable
that may be provided for resource cleanup
when the user of the buffer calls its.close()method.progress: an optionalcs.Progress.progressinstance
to which to report data consumed frominput_data;
any object supporting+=is acceptableClassFDIterator(_Iterator)An iterator over the data of a file descriptor.Note: the iterator works with an os.dup() of the file
descriptor so that it can close it with impunity; this requires
the caller to close their descriptor.MethodFDIterator.__init__(self, fd, offset=None, readsize=None, align=True):
Initialise the iterator.Parameters:fd: file descriptoroffset: the initial logical offset, kept up to date by
iteration; the default is the current file position.readsize: a preferred read size; if omitted thenDEFAULT_READSIZEwill be storedalign: whether to align reads by default: if true then
the iterator will do a short read to bring theoffsetinto alignment withreadsize; the default isTrueClassFileIterator(_Iterator, SeekableIteratorMixin)An iterator over the data of a file object.Note: the iterator closes the file on__del__or if its.closemethod is called.MethodFileIterator.__init__(self, fp, offset=None, readsize=None, align=False):
Initialise the iterator.Parameters:fp: file objectoffset: the initial logical offset, kept up to date by
iteration; the default is 0.readsize: a preferred read size; if omitted thenDEFAULT_READSIZEwill be storedalign: whether to align reads by default: if true then
the iterator will do a short read to bring theoffsetinto alignment withreadsize; the default isFalseClassSeekableFDIterator(FDIterator, _Iterator, SeekableIteratorMixin)An iterator over the data of a seekable file descriptor.Note: the iterator works with anos.dup()of the file
descriptor so that it can close it with impunity; this requires
the caller to close their descriptor.ClassSeekableFileIterator(FileIterator, _Iterator, SeekableIteratorMixin)An iterator over the data of a seekable file object.Note: the iterator closes the file ondelor if its
.close method is called.MethodSeekableFileIterator.__init__(self, fp, offset=None, **kw):
Initialise the iterator.Parameters:fp: file objectoffset: the initial logical offset, kept up to date by
iteration; the default is the current file position.readsize: a preferred read size; if omitted thenDEFAULT_READSIZEwill be storedalign: whether to align reads by default: if true then
the iterator will do a short read to bring theoffsetinto alignment withreadsize; the default isFalseClassSeekableIteratorMixinMixin supplying a logical with aseekmethod.ClassSeekableMMapIterator(_Iterator, SeekableIteratorMixin)An iterator over the data of a mappable file descriptor.Note: the iterator works with anmmapof anos.dup()of the
file descriptor so that it can close it with impunity; this
requires the caller to close their descriptor.MethodSeekableMMapIterator.__init__(self, fd, offset=None, readsize=None, align=True):
Initialise the iterator.Parameters:offset: the initial logical offset, kept up to date by
iteration; the default is the current file position.readsize: a preferred read size; if omitted thenDEFAULT_READSIZEwill be storedalign: whether to align reads by default: if true then
the iterator will do a short read to bring theoffsetinto alignment withreadsize; the default isTrueRelease LogRelease 20240201:
CornuCopyBuffer: read1() method shorthand for read(..,one_fetch=True).Release 20230401:CornuCopyBuffer.promote: document accepting an iterable as an iterable of bytes.CornuCopyBuffer: new readline() method to return a binary line from the buffer.CornuCopyBuffer.promote: assume objects with a .read1 or .read are files.Release 20230212.2:BREAKING: drop @chunky, superceded by @promote for CornuCopyBuffer parameters.CornuCopyBuffer: subclass Promotable.Release 20230212.1:
Add missing requirement for cs.gimmicks.Release 20230212:
CornuCopyBuffer: new promote() method to promote int,str,bytes, also assumes nonspecific iterables yield byteslike instances.Release 20211208:
CornuCopyBuffer.init: bugfix for self.input_data when copy_chunks is not None.Release 20210316:New CornuCopyBuffer.from_filename factory method.New CornuCopyBuffer.peek method to examine the leading bytes from the buffer.Release 20210306:CornuCopyBuffer.from_file: improve the seekability test, handle files with no .tell().CornuCopyBuffer.from_bytes: bugfix: set the buffer.offset to the supplied offset.CornuCopyBuffer.from_bytes: queue a memoryview of the supplied bytes.Release 20201102:
CornuCopyBuffer: new optionalprogressparameter for reporting data consumed frominput_data.Release 20201021:CornuCopyBuffer.from_file: changes to the test for a .seek method.CornuCopyBuffer.read: call extend with short_ok=True.CornuCopyBuffer.from_fd: record the fd as .fd, lets users os.fstat(bfr.fd).New CornuCopyBuffer.as_fd method to return a readable file descriptor fed from the buffer by a Thread, intended for feeding subprocesses.New CornuCopyBuffer.iter(maxlength) to return an iterator of up to maxlength bytes.CornuCopyBuffer.init: new "close" parameter to release resources; new CornuCopyBuffer.close method to call this.Some small fixes.Release 20200517:CornuCopyBuffer.skip: bugfix sanity check.FileIterator: do not close the supplied file, just set self.fp=None.Improve EOFError message text.Release 20200328:CornuCopyBuffer.takev: bugfix adjustment of buf.offset, was not always done.CornuCopyBuffer.getitem: add slice support, note how expensive it is to use.Release 20200229:New CornuCopyBuffer.byte0() method consuming the next byte and returning it as an int.CornuCopyBuffer.takev: bugfix for size=0, logic refactor.CornuCopyBuffer: new .selfcheck method.Release 20200130:
CornuCopyBuffer.skip: bugfix adjustment of skipto for already buffered data.Release 20191230.1:
Docstring updates. Semantic changes were in the previous release.Release 20191230:CornuCopyBuffer: accept a size of Ellipsis in .take and .extend methods, indicating "all the remaining data".CornuCopyBuffer: refactor the buffering, replacing .buf with .bufs as an array of chunks;this enables support for the new .push method and reduces memory copying.Release 20181231:
Small bugfix.Release 20181108:
New at_eof() method. Python 2 tweak to support incidental import by python 2 even if unused.Release 20180823:
Better handling of seekable and unseekable input data. Tiny bugfix for from_bytes sanity check.Release 20180810:Refactor SeekableFDIterator and SeekableFileIterator to subclass new SeekableIterator.New SeekableMMapIterator to process a memory mapped file descriptor, intended for large files.New CornuCopyBuffer.hint method to pass a length hint through to the input_data iteratorif it has ahintmethod, causing it possibly to make a differently sized fetch.SeekableIterator: newdelmethod calling self.close() - subclasses must providea .close, which should be safe to call multiple times.CornuCopyBuffer: add support for .offset and .end_offset optional attributes on the input_data iterator._BoundedBufferIterator: add .offset property plumbed to the underlying buffer offset.New CornuCopyBuffer.from_mmap to make a mmap backed buffer so that large data can be returned without penalty.Assorted fixes and doc improvements.Release 20180805:
Bugfixes for at_eof method and end_offset initialisation.Release 20180726.1:
Improve docstrings and release with better long_description.Release 20180726:
First PyPI release: CornuCopyBuffer and friends.