franciscojunior/Npgsql2 — pull request #52 by glenebob (2013-09-11 20:04:47+00:00)
Title: Lots of optimization.

Francisco,

Here is where I am currently at with Npgsql. This branch is a fork and continuation of PR 50, so if you merge this, you will get everything in 50 as well. If you merge 50, I will need to marge on my end and send up a new PR for these changes.

This last commit was inspired by Rungee's work on bytea text decoding.

The text backend type codecs now deal with byte[] to and from PG, not string. This allows a potentially huge memory savings in things like bytea, since it can be encoded directly to/from byte[], rather than using strings (which are UTF16) and then UTF8 encoding/decoding. Much much more efficient now. Text array encoding is also now efficient in the same way.

In order to keep that new efficiency, I also had to re-write the query command text generator, writing directly to byte[]. That allows text parameter values to be concatenated without first UTF8 decoding them (and then later UTF8 re-encoding).

The only text encoded data that actually uses anything above the ASCII character range is text itself. In theory, all encodings besides raw text could go directly to/from byte[]. However, I put together an Int2ASCII() function in test (writing directly to byte[]), and no matter how I optimized it, I could not get it much faster than UTF8Encoding.GetBytes(Int32.ToString()), and since the space savings on an individual Int32 is negligible, I left it out.

Some of the optimizations from PR 40 (bind optimizations) are also in here. See my latest comment on that PR.

Note that I did not optimize the bytea text decoders. Rungee did a great job with that. Instead, after this all gets merged, I plan to modify Rungee's optimizations to read directly from the byte[] data from PG, which will make it even more efficient. In the mean time, Rungee's implementation will need to use a UTF8Encoding.GetString() to get the string data it currently needs. Switching it to byte[] will be very simple for me to do.

There's more work to do on this. Bit string I think could be rewritten for use directly on the byte[]. I also noticed a huge opportunity to optimize NpgsqlCommand.GetParseCommandText(). But I kind of want to slow down and get synced back up with your master branch before doing any more work. This PR has gotten really massive.

Welcome back, and happy testing!

-Glen