openmicroscopy/openmicroscopy — pull request #3264 by jburel (2014-12-09 21:36:05+00:00)
Title: Support for collapsing unique light paths (rebased onto develop)

This is the same as gh-3261 but rebased onto develop.

----

Logic and unit test cases for collapsing unique light paths. Test cases were also expanded to cover all other existing collapse logic. Some existing, but old and non-functional, test cases have been ported to TestNG and the entire suite of `ome.formats.*` unit tests added to the server unit test suite.

Due to the amount of white space changes that were required, adding `?w=1` to the end of the GitHub "Files Changed" page may be beneficial during review.

Manual testing has been performed:
* Imported one or more Plates with `LightPath` metadata populated (Flex in particular is useful here)
* Confirmed that `LogicalChannel` and `LightPath` reduction has taken place
* Confirmed that the Plates can be deleted
* Confirmed that the Plates can be be moved between groups
* Testing also took place in the presence of a backported version of #3228

`LightPath` and `LogicalChannel` reduction can be confirmed by examining the import log for a sufficiently small number. For example:

```
2014-11-07 11:15:50,320 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique objective settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique light settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique detector settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique light paths: 2
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique logical channels: 2
```

Example datasets contained 12 fields, 384 wells and 2 channels resulting in 9216 unique logical channels per Plate. Completely broken on import. With a 2GB heap, takes ~4.5 hours just to get to pixel data processing:

```
...
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique objective settings: 1
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique light settings: 2
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique detector settings: 0
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique logical channels: 9216
...
```

Reduced version with just one Well's worth of `.flex` files:

```
...
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique objective settings: 1
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique light settings: 2
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique detector settings: 0
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique logical channels: 24
...
```

After reduction with just one Well's worth of `.flex` files:

```
...
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique objective settings: 1
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique light settings: 2
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique detector settings: 0
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique light paths: 2
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique logical channels: 2
...
```

Total time to import with a 2GB heap is now under <60 minutes.

```
_____________________
< I love light paths! >
---------------------
\ ^__^
\ (**)\_______
(__)\ )\/\
U ||----w |
|| ||
```