openmicroscopy/openmicroscopy — pull request #2767 by joshmoore (2014-07-08 05:42:04+00:00)
Title: Event log queue (rebased onto develop)

This is the same as gh-2639 but rebased onto develop.

----

Intended as `PersistentEventLogLoader Mk II`, `ome.services.eventlogs.EventLogQuery` works around a number of issues spelled out in [11948](https://trac.openmicroscopy.org.uk/ome/ticket/11948) and elsewhere. Rather than loading a single event log at a time, number logs (by default 1 million) are loaded *without duplicates* and further attempts to add a log which is already cached will not lead to an extra indexing of the object.

Currently, this has only be implemented for the `FullTextIndexer` but the same logic could be applied to the `PixelDataHandler`, and in fact, the two might could coexist in the same JVM. This PR will need a number of different types of review and should probably stay open for some time to guarantee that any quirks have been worked out. It builds on top of the add-metrics branch PR (gh-2581) and so should be able to provide reasonably accurate feedback of what it's doing (JMX/SLF4J/Graphite).

If necessary, a server installation can rollback to the previous `persistentEventLogLoader` via `bin/omero config`.

Other improvements here:
* Shutdown time of `Indexer-0` is **significantly** improved.
* A number of new CLI `reindex` commands are now available, e.g. `--reset` and `--sequential`.
* `reindex` is configurable in terms of `--merge_factor`, `--ram_buffer_size`, `--lock_factory` and `--mem`. (See `bin/omero admin reindex -h` for more)
* Indexer logging should be much user friendlier, both from the CLI and in the background.

Things needing work:
* Docs, as always
* Configuration properties and their defaults (especially for logging)
* Signal handling of `fulltext.Main` needs to be reviewed, especially on Windows. A **job** may be necessary for testing these features. /cc @sbesson

See also:
* http://trac.openmicroscopy.org.uk/ome/ticket/11950