New build scripts

With the paradigm for building changing away from compiling MM ourselves
(phew), I've started work on these new build scripts.

build-do-setup.sh is the successor to the transient setup.sh; this fetches
an MM nightly build installer and installs it.

build-mm-prereqs.sh compiles MMCore (C++ and Java) and mmstudio, in case
we make changes to these that we want to keep around. mmstudio now has a
(largely nonstandard) pom.xml, letting it be built with maven.

build-ant.sh and build-maven.sh use maven or ant (guess which uses which!)
to build SPIMAcquisition. Maven is the default.

build-fragments.sh is a function library; pay no attention to the script
behind the curtain.

build-new.sh is the main script; it uses the others to compile a
(hopefully) working version of MicroManager + Fiji + SPIMAcquisition.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
New build: Create SPIMAcquisition autostart script

Our autostart script ceased working a while ago, with a revamp of the
plugins system in Micro-Manager. The visibility thereof is much more
carefully managed now, so the script takes an alternative approach: It
waits until the MM menu is established, then runs doClick on the
appropriate JMenuItem.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Remember last output directory

A minor convenience feature.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Play nicer with missing stages

When MM first starts, the user won't have a configuration -- if the plugin
autostarts as it now is meant to, it shouldn't expect the stage to exist.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
AcqOutputHandler: Provide more useful information

Originally output handlers were not informed of the current timepoint/view
index. Ideally they wouldn't need to be, but for some of the more
structured output modes (specifically OME-TIFF), the information is
extremely useful.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
OME-TIFF output handler rework

This particular issue has been rather painful, all things told truly.

I decided to change the way the program outputs OME-TIFFs: now, each
view/timepoint is its own image, matching the filesystem structure. The
upside is they can have varying plane counts, supporting continuous
acquisition. The downside is different time points of the same view aren't
considered related (strictly speaking).

The OME-XML specification isn't totally clear (to me, anyway) on how these
things relate. For instance, the 'plane' elements that reference the
actula 2D image data are assigned time and Z indices that should relate
them to each other. However, since two such elements from different times
are now under separate parent 'image' elements, things like the bioformats
importer don't recognize them as the same.

Unfortunately the two behaviors are mutually exclusive: An image cannot
consider two blocks with varying Z counts to be the same position. In a
future commit I hope to restore the 'correct' behavior but only in the
case of well-structured (constant dimensions) data. And perhaps if it
becomes possible, I can implement the 'correct' behavior for
poorly-structured data by multidimensional padding.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Better integration of continuous acquisition

Previously, continuous acquisition wasn't possible with writing to disk.
This was largely because OME-TIFF can't support ragged/poorly-structured
data, which continuous acquisition rather implies: because thread
scheduling is nondeterministic, image capture rate varies, leading to
differently-sized Z stacks for the same span of Z coordinates.

A previous commit enabled the OME-TIFF handler to output such data (at the
expense of the XML structure not appropriately reflecting the data
itself), so this capability has been restored. With a few other tweaks,
continuous acquisition works with the OME-TIFF handler.

There are still serious drawbacks -- the framerate, for instance, is
apalling. It may be that our motors are too fast (over a 1500-um span, we
get ~50 images). As I mentioned in the OME-TIFF commit message, the data
structure isn't correct (the stacks aren't considered related by the
Bio-Formats Importer). Anti-drift does not work with continuous mode yet.
These issues are to be addressed in future commits.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Redundant Asynchronous Output

WIP: Doubly asynchronous output

Unfortunately, async output is not always enough. In particular, overnight
time-lapses have apparently run into troubles, according to Pete. (And
worse, this was on a 64-bit computer with 24GB RAM.

So I'm changing strategy. Rather than limit the image queue on
construction, I'll monitor the memory available according to IJ and decide
whether to (a) enqueue a full image, (b) write the image to a temporary
file and enqueue that path, or (c) block and wait for enough space to do
(b).

Obviously, (c) is the bad choice. In the past we've avoided that using
only option (a), then falling back to (c) if there's a problem.

Option (b) sounds expensive, but it's actually not too bad -- IJ is very
quick about writing out a TIFF. It's slower than storing in memory, but it
may still buy enough time to keep acquiring images.

In general, these changes should improve the async's ability to cope with
long-term acquisitions. The thresholds are raised (from a hard line of
0.5*available memory, to 0.6*max for straight enqueue and 0.8*max for
ram-disk-ram enqueues), and using the current available memory should
allow GC to help more.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>

Doubly-asynchronous output major rework

(Need to squash this into the other commit and move that one forward.)

I changed the asynchronous paradigm pretty signficantly. Rather than
explaining the changes, I'll just explain the new handler:

There are three tweakable parameters: The memory quota, the disk quota,
and the GC aggressiveness. The handler stores slices in memory until
exceeding the memory quota, at which point it starts to write single-slice
tiffs to a temporary directory in the output directory. This continues
until the disk quota is met, at which point it starts desperately
searching for options.

While running, if the number of freed slices : total memory slices (queued
+ freed) exceeds the GC aggresiveness threshold, the handler forcibly
cycles the garbage collector twice and resets the freed slice count. This
isn't exact by any means -- GC cycles in between forced runs won't reset
the freed slice count (TODO: phantom reference queue?), and we're tracking
slices instead of bytes, but it's effective at holding off the
disk-writing. However, the GC won't be run if the handler is mid-stack
(slowed), to prevent interfering with the imaging.

The inquisitive can try playing with the settings to find the optimum
balance. I recommend a memory quota around 80% (higher than 85% triggers
runtime garbage collections, while higher than 90% risks OoM errors), a
disk quota around 90% (this can be pushed pretty high, though 100% will
cause 'disk full' errors), and a GC aggressiveness of around 50%.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
SPIMAcquisition: Decimal size estimate

The whole-number size estimate was appealing, but there's a lot of room in
GB for wiggle. Show a little more detail.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
SPIMAcquisition: Update POM

We were using a pretty old version of pom-scijava. I changed the version
indicator to 'LATEST' because I'm lazy. (I realize this isn't usually
recommended practice, but in the case of SPIMAcquisition, I expect the
benefit -- being immediately aware of API changes -- may outweigh the
dangers.)

ctrueden also identified a few other issues with the POM; some old version
numbers, unused dependencies, and used-but-not-directly-referenced
dependencies. These have been corrected.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Switch output/anti-drift to using TaggedImage

As convenient as ImageProcessors might be, MM seems to prefer TaggedImage,
and there's no good reason we can't use this format. MDUtils provides
well-defined access to the metadata. Additionally, this eases the
implementation of ProgAcq as an AcquisitionEngine2010.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Improve flexibility of AcqRow

If we wish to have any hope of implementing the IAcquisitionEngine2010
interface, we need to support arbitrary Z slice sets.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
Continuous mode: Massive framerate increase

Some profiling revealed that the JNI calls to get stage positions for
metadata (which aren't valid in continuous mode anyway) were slowing down
the acquisition by quite a bit.

By moving the continuous acquisition code out of a thread and into the
main acquisition body, and minimizing the potential JNI calls, we can
essentially keep the MM native buffer empty, draining the images straight
to output -- provided the output can accept them fast enough.

Notes:
- The buffer can overflow pretty easily and there is no indication this
  has happened. There is a MM API call that can detect this; a future
  patch will include it.
- Stage position metadata is unavailable/invalid in continuous mode. I
  don't see a good way to obtain it from Java. I will experiment with
  changes in CMMCore to automatically include that information.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
ProgrammaticAcquisitor: Add AcqStatus object

This object is meant to provide some more detailed control over the
acquisition process. It contains a status flag, updated by the
performAcquisition method, which can be checked externally to monitor the
acquisition. It also contains a pause flag that will halt the acquisition
temporarily (required by IAcquisitionEngine2010). (This feature isn't very
well-tested; the behavior may be removed at a later date.)

Note that this doesn't and won't deprecate the status callback currently
used by SPIMAcquisition.

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>
WIP

Signed-off-by: Luke Stuyvenberg <stuyvenberg@wisc.edu>