ieb/solr — Issue #72: Support indexing priority in the Queue.
Author: ieb
Created: 2011-10-27 17:36:01+00:00
State: open

The indexing Queue is a single priority queue which means that high priority indexing operations get blocked by high volume low priority indexing operations. The queue itself and the underlying stores are capable of operating at low latency and high speed. To allow the application to prioritise events related to indexing we need to be able to fast track those events through the queue with minimal latency so they are indexed correctly.

Potential Solutions.
A. 2 queue.
Events come in and are placed in a queue according to their priority. Separate threads process each queue independently.

B. 1 queue dual scan
We keep 1 queue 1 thread and scan according to priority

C. Bypass the queue
We put something in that ensures the event never gets queued and gets indexed in the request thread.

D. Something else
If there is something else, I am all ears.

Analysis.
A. has the simplicity that we can reuse the existing code by has the drawback of concurrent access to parts of the Solr J client, the source information and the Solr Server. I am less worried about the Solr server since in a real cluster we will probably run that remotely and so it will be accessed over http concurrently. I am worried about multithreadedness in indexing especially when running in embedded mode since we have been running single threaded till now and there were significant threading issues for index write operations in Sakai 2. A few non request threads is probably safe and doesn't risk server deadlocks.

B. This is ideal, but complex since the single thread would have to multi task and context switch. Its probably better achieved by A with synchonization blocks.

C. This is a non starter. In a cluster, its only possible to have 1 solr master accepting writes per core. If that solr master goes away, then the queue protects against loss of data. If the queue is bypassed, data will be lost. Thats the main reason. The second reason is that rather than having a limited number of background threads entering in the indexing operations we now have request threads. When the index needs to be optimised or anything major needs to happen in indexing cycle the request threads will be blocked. That will lock the server and consume all threads giving 503s back to the user until the operation is complete. Lucene was not designed as a high update inverted index and its only in Solr4 that soft commits allow it to be used more real time, but we still cant expect it to reliably respond to write operations in the sort of times that are required by request threads. We dont have many of them passing through the QoS filter and to have any of them block will be a disaster. The other minor issue is that some types of content can take a long time to index (vector text PDFs). If one of those is pending in the input queue inside solr then its likely it will block other operations.

WIth all of these approaches, items that take a long time to process prior to being written into the lucene index must not go into the priority queue.