openmicroscopy/openmicroscopy — pull request #3261 by chris-allan (2014-12-09 11:29:53+00:00)
Title: Support for collapsing unique light paths

Logic and unit test cases for collapsing unique light paths. Test cases were also expanded to cover all other existing collapse logic. Some existing, but old and non-functional, test cases have been ported to TestNG and the entire suite of `ome.formats.*` unit tests added to the server unit test suite.

Due to the amount of white space changes that were required, adding `?w=1` to the end of the GitHub "Files Changed" page may be beneficial during review.

Manual testing has been performed:
* Imported one or more Plates with `LightPath` metadata populated (Flex in particular is useful here)
* Confirmed that `LogicalChannel` and `LightPath` reduction has taken place
* Confirmed that the Plates can be deleted
* Confirmed that the Plates can be be moved between groups
* Testing also took place in the presence of a backported version of #3228

`LightPath` and `LogicalChannel` reduction can be confirmed by examining the import log for a sufficiently small number. For example:

```
2014-11-07 11:15:50,320 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique objective settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique light settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique detector settings: 1
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique light paths: 2
2014-11-07 11:15:50,324 INFO [ ome.formats.OMEROMetadataStore] ( main) Unique logical channels: 2
```

Example datasets contained 12 fields, 384 wells and 2 channels resulting in 9216 unique logical channels per Plate. Completely broken on import. With a 2GB heap, takes ~4.5 hours just to get to pixel data processing:

```
...
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique objective settings: 1
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique light settings: 2
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique detector settings: 0
2014-11-05 06:12:13,751 INFO [ ome.formats.OMEROMetadataStore] (.Server-10) Unique logical channels: 9216
...
```

Reduced version with just one Well's worth of `.flex` files:

```
...
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique objective settings: 1
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique light settings: 2
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique detector settings: 0
2014-11-07 05:43:23,329 INFO [ ome.formats.OMEROMetadataStore] (erver-2027) Unique logical channels: 24
...
```

After reduction with just one Well's worth of `.flex` files:

```
...
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique objective settings: 1
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique light settings: 2
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique detector settings: 0
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique light paths: 2
2014-11-07 05:52:44,408 INFO [ ome.formats.OMEROMetadataStore] (l.Server-3) Unique logical channels: 2
...
```

Total time to import with a 2GB heap is now under <60 minutes.

```
_____________________
< I love light paths! >
---------------------
\ ^__^
\ (**)\_______
(__)\ )\/\
U ||----w |
|| ||
```