clarencen/lshift — Issue #301: Handling large numbers of entries
Author: clarencen
Created: 2011-11-14 22:42:26+00:00
State: open

Currently, the HibernateVersionCorrelationStore retrieves ids from the indexer, and then sends them into the database via a series of constraints (one per id). There is a high likelihood that this is going to hit database limits, resulting in queries being rejected when there are lots of entities in the system.

To correct this, we're considering making the Lucene index the primary store for correlations. Correlations would no longer live in the database, and would be directly stored (and retrieved) from the index.

To progress this, the plan is:

1. Create a performance benchmark test loading very large numbers of entities to confirm the failure scenario;
2. Eliminate the HibernateVersionCorrelationStore, and make the LuceneIndexer implement the VersionCorrelationStore functionality directly;