New custom stream classes replacing BufferedStream and MemoryStream

Hi all,

Per discussion with @roji, I implemented some Stream classes to replace BufferedStream and MemoryStream, and got them working in Npgsql.

This is really preliminary work, and just the start of a much bigger project that I don't have time to finish, but i think it provides part of the basis needed for Npgsql to really take control of its buffer management.

i did not implement a buffer pool, which may prove to be another important aspect of this project, but a buffer pool is really quite trivial to write, and adapting NpgsqlMemoryStream to it would also be trivial.

The hard, tedious work ahead will lie in refactoring Npgsql to extend and use these new streams to eliminate a lot more allocations and copy operations, partly by taking advantage of the fact that we can have full access to their buffers.

-Glen

OK I think that addresses the comments.  Thanks for the great feedback
guys. Between your comments and some quality time spent staring at the code
and testing, I was able to make quite a few improvements.

I still have a test failing on my 9.3 PG instance. Can you guys try the
tests on a 9.4 instance and see what you get?
EntityFrameworkBasicTests.TestComplicatedQueriesWithApply is being rejected
by PG with 'syntax error at or near "SELECT"'.

-Glen


On Sat, Nov 1, 2014 at 8:21 PM, Glen Parker <glenebob@gmail.com> wrote:

> On my machine (Windows 8) it's returning 6 for UTF-8, which seems to agree
> with the standard. OTOH, it seems that there are not yet any character sets
> defined that require that many bytes. I'm not sure how we should proceed
> here. I don't want to hard code anything here, as standards and character
> sets evolve.
>
> -Glen
>
> On Thu, Oct 30, 2014 at 6:40 PM, Emill <notifications@github.com> wrote:
>
>> In Npgsql/NpgsqlStream.cs:
>>
>> > +    {
>> > +        protected byte[] readBuffer;
>> > +        protected int readBufferCapacity;
>> > +        protected int readBufferPosition;
>> > +        protected byte[] writeBuffer;
>> > +        protected int writeBufferPosition;
>> > +        protected bool performNetworkByteOrderSwap;
>> > +        protected Encoding textEncoding;
>> > +        protected int maxBytesPerChar;
>> > +
>> > +        public NpgsqlStream(bool performNetworkByteOrderSwap, Encoding textEncoding)
>> > +        {
>> > +            this.performNetworkByteOrderSwap = performNetworkByteOrderSwap;
>> > +            this.textEncoding = textEncoding;
>> > +
>> > +            this.maxBytesPerChar = this.textEncoding.GetMaxByteCount(1);
>>
>> This is 1 more than the real max bytes per char for all encodings (except
>> for UTF7), which might not be want we want. For example, for UTF8, this
>> returns 4. But all chars can be encoded with at most 3 bytes. The reason it
>> returns 1 more is that they assume you might want to use the Encoder class
>> that can buffer 1 char in case of left over surrogate pairs between two
>> calls to its Convert function.
>>
>> —
>> Reply to this email directly or view it on GitHub
>> <https://github.com/npgsql/npgsql/pull/388/files#r19647703>.
>>
>
>