BaseXdb/basex — Issue #319: REPLACE very slow on databases with many documents
Author: malamut2
Created: 2011-12-15 21:39:26+00:00
State: open

For our use case, we try to keep a database with approx a million documents, all of which need to be replaced once in a while with up-to-date versions. It turns out, however, that the time needed for a single REPLACE command currently is (mostly) proportional to the number of documents in the database, yielding in processing times of several seconds per single (tiny) file in our use case which basically destroys our whole concept.

Having looked closer into the code, the problem seems to me that after each REPLACE, DocIndex.update() is called (several times, in fact), thus invalidating the path structure. On the next REPLACE command, the complete path structure is then re-built from scratch.

There should be a way to avoid this thrashing. It would be ideal if REPLACE could just update the path structure instead of invalidating it - I think in most cases, this would come down to a single add or even no change at all. Anyway, the time needed for a REPLACE should not at all depend on the number of documents residing in the database.

Do you think this could easily be fixed?