fiji/fiji — pull request #15 by larrylindsey (2013-04-26 22:33:19+00:00)
Title: Fiji Archipelago

Update to Fiji Archipelago Cluster Service plugin.

# Overview #
The Archipelago plugin provides an easy interface for exporting computation over the network to remote computers, via ssh.

## Starting a Cluster##
* From the ImageJ window, select Plugins->Cluster->Start Cluster...

### Configure the Root Node ###
* Click the Configure Root Node... button

Local Exec Root: The folder containing your local fiji (or ImageJ) executable. This field may disappear in future versions.
Local File Root: A shared network folder, if one is available. Not necessary for most jobs.
User Name: Your user name on this machine
Default Client Exec Root: The folder containing fiji (or ImageJ) on most remote clients. This field may disappear in future versions.
Default Client File Root: The matching network folder, relative to remote clients.
* Click OK

### Configure Remote Nodes ###
#### Starting them via SSH ####
Archipelago uses JSch to start remote nodes by ssh. This requires key file authentication to work because I don't want to store your password. There are two options for how this works.

The first is the SSH Shell method, which starts the remote node and uses standard io for communication. The benefit is that traffic is encrypted, and security concerns are deferred to your implementation of ssh.

The second is the Insecure Socket Shell method, which starts the remote nodes by ssh and causes them to connect to a server running on the local machine, using an unencrypted Socket for traffic. This may be faster, but should not be used on an unprotected network.

* Click Configure Cluster Nodes...
* Click Add Node...

Hostname: The hostname or ip address for this host
User name: The username for this host
Thread Limit: Allow up to this many Callables to be scheduled on this host. If set to 0, this will be set to the number of available cores.
Remote Fiji Root: The folder containing fiji or ImageJ on this host.
Remote File Root: The shared network folder relative to this host.
Shell: SSH Shell or Insecure Socket Shell
keyfile: Your ssh keyfile, as generated in Linux by ssh-keygen
executable: either fiji or ImageJ
ssh-port: the ssh port for this host

* Click OK
* When you have added all of your nodes, click OK

#### Starting them from the Keyboard ####
Instead of SSH, nodes may also be started locally, ie, by walking over to them and using a keyboard an mouse. This involves the use of an insecure server, and is not recommended unless you trust your network.

On the root node:
* Click Start Cluster
* Click Start Insecure Server

On the client node:
* Start Fiji
* Click Plugins->Cluster->Attach to Cluster...
* Enter the hostname and port of the root node, then click OK

OR

Run ./fiji --full-classpath --main-class edu.utexas.clm.archipelago.Fiji_Archipelago root.node.hostname port

The default port is 4012

## Running Code on a Cluster ##

For most use cases, a root node will run only one instance of Cluster. This is referred to by calling [Cluster.getCluster()](https://github.com/larrylindsey/fiji/blob/fiji-archipelago-security/src-plugins/Fiji_Archipelago/src/main/java/edu/utexas/clm/archipelago/Cluster.java#L853)

Cluster returns an [ExecutorService](http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html) through [Cluster.getService(n)](https://github.com/larrylindsey/fiji/blob/fiji-archipelago-security/src-plugins/Fiji_Archipelago/src/main/java/edu/utexas/clm/archipelago/Cluster.java#L1751), where n is either a float or an int. If an int, any Callable or Runnable submitted to the service is assumed to occupy n number of cores. If a float, they are assumed to use a fraction n of the available cores.

For most cases, Cluster.getCluster().getService(1) will do what you want.

Care must be taken when writing your Callable. It must implement Serializable, because it will be serialized and sent over an Object Stream to the remote node. Because of this, deep equality of any internal variables will be broken. See this [example](https://github.com/larrylindsey/fiji/blob/fiji-archipelago-security/src-plugins/Fiji_Archipelago/src/main/java/edu/utexas/clm/archipelago/example/Equality_Example.java)

An example using Archipelago to extract SIFT features may be found [here](https://github.com/larrylindsey/fiji/blob/fiji-archipelago-security/src-plugins/Fiji_Archipelago/src/main/java/edu/utexas/clm/archipelago/example/Cluster_SIFT.java)