BaseXdb/basex — pull request #280 by somedave (2011-11-18 19:13:44+00:00)
Title: Streaming insert

As I discussed in my message board post, this is an initial attempt at creating a pure streaming insertion capability. It currently only works with subsequent adds (I.e., the Add command) since an intermediate Builder is not used when first constructing a database. I've updated the Add command to use the new approach for testing, but left the old code commented in-place.

My initial results are promissing, yielding 20% - 30% performance improvements for large XML files. However, it doesn’t appear to help in all cases which greatly confuses me (I was expecting an across-the-board performance improvement). It appears as though the original “Parser -> Builder -> temp Data -> target Data” is faster than the new “Parser -> Builder -> target Data” for smaller documents. I’m going to work on some profiling and benchmarking to try and figure out why the intermediate Data instance actually helps in these cases. I suspect it has something to do with the kind of caching the InsertBuilder has to do since it's relying on a forward-moving stream.

Below are some notes on the code:

The InsertBuilder class is based heavily on Data.insert(). This creates a lot of code duplication and might result in maintenance problems. The OO purist in me would prefer to find a way to remove the duplication. I see two ways this could be done. The Data.insert() method that takes a source Data could actually use the InsertBuilder along with a Parser that fires events by traversing the source Data. Alternativly, some kind of middle-tier class (perhaps DataInserter) could contain the logic for data insertion in a way that both Data.insert() and InsertBuilder could make use of it.

There is some duplication between the base Builder and InsertBuilder. For example, Builder has a pre stack that gets populated internally and InsertBuilder also has a pre stack that gets populated seperatly. It’s possible they both have the same behavior, but it was easier on this first time around to try and isolate the logic from Data.insert() as much as possible inside the InsertBuilder class. Improvements may include taking advantage of native Builder functionality where duplicate behavior exists.

This next note is actually a bug with the base Builder class. The declaration for the abstract Builder.addElem() was incorrect. It had parameter names and descriptions in the wrong order. Changed it to match call site and overrides.

Also fixed what I believe to be a bug in Data.insert() with the calculation of dis - it was commented with ([LK] if ipar == -1 wrong dis is calculated?). Turns out I had already found this to be a problem and had fixed it in my previous .NET code. I also fixed a problem with the calculation of par for root node inserts - it had similar problems.