BACKGROUND
multi-dimensional images are integral to biomedical research with many large scale imaging initiatives now in place to capture image data with genomic-scale coverage.
these can be high resolution two- and three-dimensional  and time varying images from the cellular through to whole organism level of resolution  <cit> .
this data acquisition has been matched by the development of image archive and database systems to support the curation and query of the data  <cit> .
most of these systems will provide a number of visualisation options but in general these require the download of the full data-sets to be visualised using applications on the users’ workstation.
in addition there are now a number of image databases that include a standard spatio-temporal reference framework onto which large volumes of data are spatially mapped to enable cross-comparison, query and analysis.
visualisation of these mapped data-sets could imply massive download of data beyond the local disk capacity and available bandwidth.
in this context therefore the ability to access 3d image objects over a wide-area network such as the internet is imperative.recent work has demonstrated the clear advantages of tile-based image transmission and many zoom-viewers have been developed for example by google maps , zoomify  and openzoom .
the key benefit is that the client application only requests enough data to display at the resolution required and therefore the data download scales in proportion to the view-window size which is in most cases constant.
in addition the tile-based approach can benefit from data-compression, asynchronous download and the use of caching which makes very effective use of the network and provides a fast response for the user.
some solutions involve the internet imaging protocol  <cit> , which is an open protocol that provides fast tiled delivery of large images through a multi-resolution image representation.
iip has been used in tele-pathology and educational archives  <cit> .
it allows a region of interest to be selected at any zoom level and provides efficient image transmission.
an iip server has been developed  as an open-source resource  <cit> .
however, iip  may only be used with 2d images and a similar presentation method for 3d objects did not exist.
therefore we have developed extensions to the iip protocol, which we call iip3d, and have implemented an open-source server to deliver this service based on the highly efficient woolz image processing library  <cit> .
in addition we have developed a number of browser-based applications for specific projects which we use to illustrate the capabilities of the system.
the importance of virtual slicing systems for remote access of images was previously noted  <cit>  and the iip protocol was identified as a suitable interface for independent client-server applications.
however the availability and flexibility of these imaging systems was limited by the proprietary and costly nature of existing implementations.
current image servers, such as brainmaps.org, are able to deliver 3d data although they provide only predefined 2d sections.
glatz-krieger et al.
<cit>  consider virtual slices only in the original focal planes of the biological material in the context of a 2d microscope slide.
in this paper, we cut an arbitrary virtual section from the digitised 3d model.
such sectioning software exists, e.g.
as commercial applications such as e.g.
amira  <cit>  or open-source projects such as slicer  <cit> , imagej  <cit>  or our own applications mapaint <cit>  and the java atlas viewer, jatlasviewer viewers  <cit> .
there are also online java applications such as neuroterrain  <cit> .
java is often selected for compatibility and platform independence, but sometimes falls short of this in practice.
tile based image delivery, that transmits the target as smaller image blocks, is known from commercial web applications such as google maps.
this runs in any web browser and does not require additional software or an applet.
we have developed a server based on the open iip protocol that can deliver arbitrary sections through large scale 3d image data providing a very fast and efficient display that can be accessed with standard browsers.
the visible human project has generated several internet based image servers and clients.
the epfl server  <cit>  is the most similar to ours.
it is a high throughput parallelised sectioning server using a fastcgi  web interface.
however it does not allow tiled requests and has a proprietary protocol.
in addition, for the delivery of section data, the epfl server requires a high performance cluster.
in contrast, the iip3d server will run on standard linux-based servers, is open source and has been tested under linux, mac os x and windows.
to supplement the viewing of arbitrary sections through 3d image data we have developed clients which can visualise any number of image layers including indexed graphical overlays.
many “atlas” views of data include the display of graphically defined regions that correspond to a segmentation of the image space.
typically these atlas regions are constrained in number  and constrained spatially not to allow overlaps.
in contrast we have developed an index image that can have an arbitrary number of regions with any combination of overlapping regions and these are provided as an indexed overlay with complete control over the colour and opacity of the individual regions.
in summary, we have extended an open protocol and associated tools which allow fast presentation of 3d volumetric data as arbitrary cut sections delivered at the level of detail and localisation requested, in the form of compressed tiled images.
this system is open source, generic and integrable with other functionalities.
this extended protocol is iip3d and in the next section we present the implementation, followed by details of the server and client software.
we present results of performance testing with multiple simultaneous access and show how the client software can be utilised for a number of atlas-based and large scale volume image delivery.
implementation
woolz image representation
core to this work is the image processing library known as woolz <cit> .
woolz is unusual in that it has separate data structures for the spatial domain of an image object and the image values, which can be  <dig>   <dig>   <dig> bit grey values, rgb α or float/double.
the domain of an image object is simply the region  of 2-d or 3-d space over which the image is defined.
this form of record is particularly appropriate for biomedical images, as it allows a compact representation of images through not storing background values outside of the foreground object of interest and it encodes arbitrary regions of space  without the need for image values.
internally woolz represents a 2d domain as a number of line segments or intervals for each line for which there is image information.
in this way an arbitrary region of 2d space can be defined in a very compact form and binary operations can be reduced to a series of interval comparisons which are very efficient for binary-set, morphological and labelling operations.
a 3d domain is simply a planewise stack of 2d domains and null entries for planes which are entirely outside of the domain.each woolz object has its own coordinate system defined with respect to a global origin of all objects and three coordinates kl and p for the columns, lines and planes respectively.
the objects have uniform sampling and real world coordinates are achieved through a voxel size  which is stored in the domain.
because objects can be located at any coordinate location, including negative values, the domain and value table are defined relative to the overall image bounding box which allows for very simple re-location of image data.
note conversion from woolz to more standard rectangular domain based image formats may lose this information.
converters exist to transform woolz image objects to a wide range of other formats including nifti, vtk, amira and standard tiff-stacks.
large woolz objects
a recent extension of woolz allows operations on large objects  that do not fit in system memory or would take many minutes to load from disk.
it uses memory-mapped files, which reside on the external memory and are loaded on-the-fly block by block as the system requires it.
3d memory mapped objects are represented as a domain and grey value pair, however the grey values are stored in fixed size cuboidal blocks, accessed through a lookup table.
the image value subdivision into cubes is similar to other solutions  <cit>  but uses memory mapping and file-system block alignment.
these block sizes are integer multiples of the file system blocks enabling very efficient read and write operations.
with solid-state disks the i/o rate is similar to data that resides on the main memory.
cutting arbitrary sections
we specify an arbitrary section through the data volume as a plane perpendicular to the line-of-sight.
this line is determined by three angles of rotation, two to establish the line direction and the third the orientation around that line.
these correspond to pitch, yaw and roll and can be defined in terms of the standard euler angles  <cit> .
the rotation centre is the “fixed point” and the section plane is set by the distance along the view direction.
the details of the transform are published  <cit> , are given in the mapaint technical report  <cit>  and are illustrated in figure  <dig>  it is defined in terms of the fixed point f, angles pitchyaw and rollscale and distance d. the viewing transform can be represented as an affine transform where the section plane is the plane of constant z″ = d. by representing the transform in this fashion is straightforward to include any further re-scaling arising from non-isotropic voxel sizes.
these underlying coordinate transformation methods have been extensively used for developing the emouseatlas models and gene-expression database and are part of the essential data captured as part of each submission.
convenient navigation with fewer free parameters is provided by a number of standard viewing modes defined in table  <dig>
iip3d server
the iip3d server is c++ software, based on a gnu gpl implementation of the iip server by pillay and pitzalis  <cit> .
this is a fastcgi  web server module that is managed by the web server  or as a standalone application and is connected by the web server as requests arrive.
the full technical detail of the woolz extensions of the iip was provided earlier <cit> , here we present a limited set of iip3d commands.
iip3d request
the iip3d server requests are parametrised http requests and typically have four components: server address: identifies the iip3d server.
here we use the http protocol and therefore the address is the url of the fcgi application e.g.
.
resource specifier: the second component of a request defines the image resource from which tiles are requested.
this for pyramidal tiffs is , while for 3d woolz objects is the command.
view parameters: specifies the parameters of the sectioning plane  and the image return  as well as setting for overlay colours and opacities.
information request: defines the specific information required from the object and sectioning plane set.
this can be a tile, a full section or parameters of the object or sectioning plane .
the supported image request commands are similar to the iip <cit> : returns the full image ; while and the jpeg-, png-compressed and uncompressed tiles.
iip3d also supports png format with lossless compression and alpha channel, both crucial for painted domain or textual overlays.
3d browsing is essentially the selection of a section by setting the various parameters defined by the , , , and commands.
the iip3d protocol also supports the viewing modes from table  <dig> which given pitch and roll will define yaw.
in addition the iip3d server can provide information on the 3d object, with the available queries listed in table  <dig>  these include information about the 3d image volume, coordinate transforms between the displayed tiles and the original 3d image, image value and true voxel sizes.
image overlays
in a number of atlases, the histological image may have overlay layers presenting for example anatomical domains or gene expression patterns.
these are analogous to satellite images overlaid with street map or traffic information.
iip3d provides this functionality with the select () command and uses the compound woolz object format to capture the set of domains.
this is an array of independent woolz objects stored in a single file and each domain can be accessed by selecting the corresponding index.
any number of such indexed domains are allowed and there are no spatial restrictions, the domains can overlap within the context of the viewed section and will be displayed as a set of overlapping colours.
the most general syntax of is where is the selected objects index, while , , , are red, blue, green and alpha channels specifiers.the rgb components paint with the chosen colour the selected domain, or for grey/colour objects filter the image selecting only the desired colour components.
simplified syntax allow  <dig>   <dig> and  <dig> parameters with ; and , with  <dig> as the default for the omitted parameters.
grey level or domain components are either independently selectable or stacked in an arbitrary order to generate grey level images, semi-transparent mask images, or a combination.
a sequence of arbitrary number of commands specifies the composition order with the alpha channel information to determine layer visibility.
the requests the index or image value at query point allowing augmented display  in the viewer application.
uniquely the compound object based representation allows multiple overlaying domains to be displayed and queried which implies that the index query will in general return a list.
caching
faster image delivery is supported by three caching levels.
as in the original iip server, the iip3d server caches the requested tiles.
if multiple users access the same data, or it returns to a previously visualised region then the tiles are not recomputed but fetched directly from the cache.
3d images are often large, and the disk-read time is a significant overhead.
therefore woolz image objects once read are cached for further operations.
however, for very large, memory mapped objects only the domain is cached, and not the values.
sections transforms which provide a fast look-up-table encoding of the affine transform are also cached.
extreme data sizes
biomedical images may easily be many gb in size, for example the visible human image is about  <dig>  gb and data sets for electron-microscopy data now reach many terabytes  <cit> .
most current desktop workstations do not have sufficient memory to load such images and are incapable of being upgraded to be able to do so.
even when a workstation is capable of reading such a large object into memory the time taken to transfer it from disk is usually inconvenient; with a disk based file system capable of transferring data at 40mb/s it would take at least  <dig>  minutes to read the visible human image into memory.
if the image is part of a resource then downloading significant data is very slow and copying multi-terabyte volumes infeasible.
in most cases however, only a small subset of the image data is needed for visualisation since the typical user will be restricted by screen size.
for this case we have extended the iip3d server to use the memory-mapped option for very large woolz images enabling access to images of tens or hundreds of gigabytes.
supported image formats
the woolz imaging system can convert from a range of 2d and 3d formats and are therefore directly supported by the iip3d server.
supported formats include: amira lattice, stanford density, gif, analyze, iplab, jpeg, biorad confocal, pnm, raw image data, tiff, nifti and vtk.
to get the benefit of memory-mapped option for very large volumes and the domain overlays with a global coordinate system, it is necessary to convert to woolz format data.
iip3d viewer
the iip3d viewer is a configurable web application which displays tiled images requested from a iip3d server.
it was initially based on the viewer of pillay  <cit>  but has now been substantially developed and re-organised.
the iip3d viewer uses a number of web technologies including: html/http, php/jsp, css, javascript, ajax .
a json configuration file identifies various system parameters including the location of the image and the iip3d server.
it also defines the parameters of all the layers associated with an image, including name, location of data and display order.
if a tree control is to be used , the structure and content of the tree is defined in a configuration file.
the central area of the viewer displays the visible region of an image.
tools for user interaction with the image will typically also be included.
the choice of tools and their respective position within the web page is configurable by the web page developer via another json file.
tools, see figure  <dig> for a selection, are available which allow for: changing the visible region of an image; changing the zoom level  of the displayed tiles; toggling the visibility of each layer on/off; changing the opacity of each layer; changing the colour filtering to be applied to grey-level layers ; for 3d wlz images changing the sectioning plane .
if overlaid data is available and has been defined in the configuration file, it is possible to generate feed-back information by use of a ’mouse-down’ action over the image.
this can be used, for example, to identify structure names within an image.
with a small amount of javascript coding it is possible for a web-page developer to add their own custom tools.
the browser tries to be efficient in terms of transmission bandwidth and client memory by: requesting tiles only for the visible region of an image; requesting tiles only at the current resolution; for a scrolled/panned object, only new tiles are requested; layers which are not visible are not loaded.
iip3d proxy
in the case where a large image archive it to made accessible via http requests it is often not desirable to have the web-server also act as the image server.
in addition with a large archive or high access rates the requests may need to be delegated to a number of servers.
to provide a single point of access and enable this delegation to image servers separated from the internet by a firewall, we have developed iip3dproxy, an fcgi application, which filters requests and forwards them to a set of iip3d servers.
the communication conforms to the fcgi protocol.
though it was designated to work for iip and woolz requests, it is generic and can route any fcgi request, hence it allows chaining of multiple proxies.
a multi-iip3d server architecture is shown in figure  <dig>  iip3dproxy is an independent program running on the proxy server.
the web server  forwards it the fcgi request on a configurable ip port, then the html request string is checked by iip3dproxy and if the definition string of any remote iip3d server is a substring of the request parameters then this query is passed over to the matching server.
if no correspondence was found then the request is passed to the default server.
RESULTS
iip3d server
tiled image
an example section query that consists of six tiles is shown in figure  <dig>  the requests, each returning a png tile, differ only in the ptl command with tile number sweeping  <dig> to  <dig>  while all have in common a fcgi application, woolz object definition () and sectioning image parameters .
overlays with sel
the iip3d server delivers tiled views of images and any given application can request tiles from multiple image volumes to be co-displayed as a series of layers with transparency provided by the browser display capabilities.
we have implemented a novel strategy using the iip3d command which can be used to request graphical overlays stored in an indexed object.
this process is depicted in figure  <dig> and overlay tiles with anatomy regions defined either individually or collectively can be retrieved and displayed as independent single- or multi-layer, overlay images.
the overlay request includes the colour and opacity for requested region.
a critical property of the woolz indexed object is that the domains represented are not limited by the number of bits used for the indexed image and and the regions can overlap i.e.
they are not spatially exclusive.
the woolz iip3d server provides the domain overlay as a single tiled image computed on-the-fly from the woolz object.
queries
the iip3d protocol includes extension of the query commands to provide information for image layout and user feedback.
these include the size  of a given section image, the size of a tile, the voxel size, the object volume and bounding box and the grey or colour value at a given pixel location within the section image.
for example, the voxel grey value query with respect to the 3d location  is http://localhost/fcgi-bin/wlziipsrv.fcgi?∖wlz=/objects/ts <dig> wlz∖pab= <dig> ,190∖obj=wlz-grey-value which results in the reply wlz-grey-value:73
this query can also be made with respect to the 2d location within a given tile.
performance evaluation
husz et al <cit>  demonstrated the performance benefit that is realised with a tiled representation with a frame rate of at least  <dig> fps on a standard wan and  <dig> fps to a domestic broadband.
however, evaluation under heavy load or simulated user interaction had not been studied.
here we present the results of testing a single iip3d server with up to sixty simultaneous users.
the test consisted of repeatedly running the recorded browsing pattern of a biologist user viewing a cropped visible human data set  with a data volume of  <dig>  gb.
the browsing pattern included tile requests , down-scaled full section request  and object queries , table  <dig>  the total requested data over  <dig> seconds of interactive browsing is  <dig>  mb, the tile size was  <dig> ×  <dig>  in the following we test two scenarios, first we respect the timing of server requests and second they are ignored which results in a more severe test of the server response.
in addition we also test the client requests with and without a random perturbation of the view parameters.
this is to ensure all requests are unique and we test the server response without any benefit of caching.
a series of experiments were performed with a single iip3d server and  <dig> to  <dig> clients.
the server and client computers were all equipped with dual 3ghz intel xeon  <dig> cpus,  <dig> r.p.m.
sata disks and 32gb ram and are directly connected via a dedicated  <dig> gb ethernet switch.
there was no other activity on the server, on the clients or on the network other then operating system management tasks.
for testing, we used standalone c++ executables started simultaneously on the  <dig> client nodes.
for test cases with more clients than available client-nodes, multiple concurrent instances were started on the same client node.
the timestamp of each request was recorded along with all other iip3d request parameters.
this allows replication of user browsing behaviour.
the object queried was already prefetched from the disk into the server main memory and the complete set of requests were repeated as whole series and timings captured.
the first test set takes  <dig> s, while the consecutive requires only  <dig> s, suggesting the effect of the local tile cache.
four different scenarios were evaluated with the results shown in figure  <dig>  first, we request in parallel the same data on a number of  <dig> to  <dig> clients.
the average response time increased from  <dig> s to  <dig> s. the time normalised to the number of clients, shows a constant time behaviour , confirming the linearity in figure 7a.
the timing requests were ignored in the second test scenario, and requests were started one after the other.
the server becomes heavily loaded by the simultaneous  <dig> requests.
however, the result graph shows again a linear behaviour, with  <dig>  ±  <dig> s per client response time.with its tile caching, the server avoids re-computation of tiles.
therefore, to force different section generation, we have have altered the original requests by adding a random value to the sectioning plane angle ().
again, figure  <dig>  there is a linear increase of the processing time with the number of requesters.
the average per client time is  <dig>  ±  <dig> s respectively  <dig>  ±  <dig> s. these times are comparable with the initial  <dig> s tile cache initialisation for our first test.
key times of these results are summarised in table  <dig>
the normalised figure is the average per client.
it is interesting to remark in figure 7b that the running time of the timed test sets recorded over  <dig> seconds, it is respected for the cached scenario, while for the non-cached scenario up to  <dig> clients managed to fit in the time user needed to browse through the object.
this suggests that up to  <dig> clients may be served without major performance degradation, however as is shown above the display delay increases.
client executables were started remotely from a test script.
we had no control on the exact timing of these processes neither how the request are prioritised by the apache web server, therefore occasionally “lucky” clients occur with lower execution time then the average.
note the client executable did not use any local caching therefore the benefit of the browser cache is not included with these measurements which are therefore a lower bound on the performance experienced by the user.
iip3d viewer
iip3d is configurable for the application in use.
we present the basic 3d browsing functionality, followed by the support of handling overlaid 3d image data and large databases.
we exemplify these functionality on the male visiblehuman dataset  <cit> , adult mouse muti-modality waxholm data  <cit> , hudsen developing human brain atlas  <cit> , fly brain atlas  <cit>  and the emouseatlas atlas models  <cit> .
3d browsing and multi-channel data
large volumetric data
large objects stored and visualised at high resolution may be read and served by the iip3d server.
these use the memory mapped data-structures that avoid the requirement to read the full dataset into main memory which for the larger volumes would not be feasible.
in addition for archive data of many 1000s of smaller 3d volumes it enables fast access to any image where again it is not feasible to maintain the image data in memory.
we have tested the system for image volumes ranging from  <dig> to  <dig> gb and found that the response was slightly slower but acceptable given that these are section views that can not for realised by any other system.
in figure  <dig> we show a view through a composite image comprising  <dig> full sets of the visible male 3d volume.
this results in a 3d image with  <dig> ×  <dig> ×  <dig> full colour  voxels with a data volume of about 138gb.
the memory mapped data format also allows us to routinely serve sections views from a multiple 3d image archive which now exceeds 4tb.
annotation overlay
CONCLUSIONS
we have extended a protocol standard for tile-based viewing of 2d image data to include 3d volumetric data to enable virtual re-sectioning at arbitrary orientations through the volume.
the protocol is also extended to handle index-image data giving control to the user of colour and transparency on each individual labelled region.
on the basis of this protocol we have implemented an iip3d server to deliver tiles for any requested section and a javascript/ajax viewer which provides a user-interface that will run within a web-browser.
performance tests on the server show that it can handle up to  <dig> simultaneous users without significant loss of performance as experienced by a typical user.
the server can be scaled using a series of proxies to provide the required performance for higher access rates as required.
the server code is implemented on a standard linux-based machine and does not require special a hpc facility.
we have tested this system in the context of atlas-based biological data however it is clearly applicable to a wide range of image data for which arbitrary re-sectioning is a useful visualisation.
this visualisation is typically the core requirement in basic science but also medicine where views through the data are the primary views needed by experts who want to see “the real data”.
it is complementary to recent development of in html <dig> in particular webgl which is exemplified by the google “body browser” demonstrator .
we plan to use iip3d in a future webgl based application to provide cut section image textures.
for archives of volumetric data or single very-large image volumes this server development is the only plausible mechanism to deliver views to users who do not have the required bandwidth, local storage capability or the compute capability to be able to download and view the data.
here we realise the benefit of low-bandwidth tile-based access to image data coupled with very efficient image-processing to calculate the virtual section tiles.
this technology is open-source and will run on any unix-based web-server.
the iip3d server allows access to independent software either as a standalone application or a more complex java or adobe flash viewer.
however, for flexibility and interoperability we provide a javascript api and viewer which runs in most up-to-date internet browser software, e.g.
internet explorer, safari, firefox and chrome but with some variation on behaviour arising from different ways in which the browsers have implemented standards .
through the iip3d api the 3d viewer can be integrated into complex applications such as next generation embryology <dig> - a 3d spatio-temporal framework that augments iip3d data in conjunction with a repository to deliver research and educational material.
users may add supplementary augmentation  to the 3d volume, together with texts, videos, external links.
the compound representation of multiple images  and providing them as individual or composed images allows compact representation of related data, and offers a novel 3d visualisation of overlays.
also, it has power of representing overlapping domains that standard index-image representations can not provide.
we have shown that the performance of our implementation of iip3d scales well with the number of parallel requests.
grey level or colour data is successfully augmented with domain or gene expression data.
controls, such as sliders, layer selection tool or locator resemble functionality of know tools from desktop applications.
these tools and the associated tiling code are all available as open-source.
future developments of this software include grey-level transforms such as range slicing for 16bit medical data, extension to vector and scalar data and the introduction of a standard glyph library for location and directional marking.
in the context of the client interface we will address usability by undertaking a user-evaluation study.
availability and requirements
· project name: woolz iip
· project home page:http://www.emouseatlas.org/emap/analysis_tools_resources/software/wlziip.html
· operating system: sever: linux; client: platform independent
· programming language: server: c and c++; client: javascript
· other requirements: server: woolz x.x, fastcgi library and web server with fastcgi enabled; client: internet browser must support javascript and display images
· license: gnu gpl
· any restrictions to use by non-academics: according to gnu gpl
endnotes
ahttp://www.emouseatlas.org/emap/help/compatibility.htmlbhttp://research.nesc.ac.uk/node/400
abbreviations
iip: internet imaging protocol; iip3d: 3d extended internet imaging protocol; fcgi/fastcgi: fast common gateway interface.
competing interests
the authors declare that they have no competing interests.
author’s contributions
zh designed and extended the original iip protocol, coded the iip3d server, suggested the existing compound woolz object representation for domain data and wrote the first draft of this paper.
bh and rb develop the woolz image processing library, rb developed the woolz sectioning algorithm and bh added support for memory mapped objects.
nb integrated features of existing browser to provide a unified application programmer interface .
nm added support of tree selection of anatomical hierarchy for overlaid images and provide input for viewer api standardisation.
rb formulated the underlying ideas for using the iip server for section data and managed the design and software development.
all authors read and approved the final manuscript.

 multi-dimensional images are integral to biomedical research with many large scale imaging initiatives now in place to capture image data with genomic-scale coverage. these can be high resolution two- and three-dimensional and time varying images from the cellular through to whole organism level of resolution. this data acquisition has been matched by the development of image archive and database systems to support the curation and query of the data. most of these systems will provide a number of visualisation options but in general these require the download of the full data-sets to be visualised using applications on the users’ workstation. in addition there are now a number of image databases that include a standard spatio-temporal reference framework onto which large volumes of data are spatially mapped to enable cross-comparison, query and analysis. visualisation of these mapped data-sets could imply massive download of data beyond the local disk capacity and available bandwidth. in this context therefore the ability to access 3d image objects over a wide-area network such as the internet is imperative.recent work has demonstrated the clear advantages of tile-based image transmission and many zoom-viewers have been developed for example by google maps, zoomify and openzoom. the key benefit is that the client application only requests enough data to display at the resolution required and therefore the data download scales in proportion to the view-window size which is in most cases constant. in addition the tile-based approach can benefit from data-compression, asynchronous download and the use of caching which makes very effective use of the network and provides a fast response for the user. some solutions involve the internet imaging protocol, which is an open protocol that provides fast tiled delivery of large images through a multi-resolution image representation. iip has been used in tele-pathology and educational archives. it allows a region of interest to be selected at any zoom level and provides efficient image transmission. an iip server has been developed as an open-source resource. however, iip may only be used with 2d images and a similar presentation method for 3d objects did not exist. therefore we have developed extensions to the iip protocol, which we call iip3d, and have implemented an open-source server to deliver this service based on the highly efficient woolz image processing library. in addition we have developed a number of browser-based applications for specific projects which we use to illustrate the capabilities of the system. the importance of virtual slicing systems for remote access of images was previously noted and the iip protocol was identified as a suitable interface for independent client-server applications. however the availability and flexibility of these imaging systems was limited by the proprietary and costly nature of existing implementations. current image servers, such as brainmaps.org, are able to deliver 3d data although they provide only predefined 2d sections. glatz-krieger et al. consider virtual slices only in the original focal planes of the biological material in the context of a 2d microscope slide. in this paper, we cut an arbitrary virtual section from the digitised 3d model. such sectioning software exists, e.g. as commercial applications such as e.g. amira or open-source projects such as slicer, imagej or our own applications mapaint and the java atlas viewer, jatlasviewer viewers. there are also online java applications such as neuroterrain. java is often selected for compatibility and platform independence, but sometimes falls short of this in practice. tile based image delivery, that transmits the target as smaller image blocks, is known from commercial web applications such as google maps. this runs in any web browser and does not require additional software or an applet. we have developed a server based on the open iip protocol that can deliver arbitrary sections through large scale 3d image data providing a very fast and efficient display that can be accessed with standard browsers. the visible human project has generated several internet based image servers and clients. the epfl server is the most similar to ours. it is a high throughput parallelised sectioning server using a fastcgi web interface. however it does not allow tiled requests and has a proprietary protocol. in addition, for the delivery of section data, the epfl server requires a high performance cluster. in contrast, the iip3d server will run on standard linux-based servers, is open source and has been tested under linux, mac os x and windows. to supplement the viewing of arbitrary sections through 3d image data we have developed clients which can visualise any number of image layers including indexed graphical overlays. many “atlas” views of data include the display of graphically defined regions that correspond to a segmentation of the image space. typically these atlas regions are constrained in number and constrained spatially not to allow overlaps. in contrast we have developed an index image that can have an arbitrary number of regions with any combination of overlapping regions and these are provided as an indexed overlay with complete control over the colour and opacity of the individual regions. in summary, we have extended an open protocol and associated tools which allow fast presentation of 3d volumetric data as arbitrary cut sections delivered at the level of detail and localisation requested, in the form of compressed tiled images. this system is open source, generic and integrable with other functionalities. this extended protocol is iip3d and in the next section we present the implementation, followed by details of the server and client software. we present results of performance testing with multiple simultaneous access and show how the client software can be utilised for a number of atlas-based and large scale volume image delivery. implementation woolz image representation core to this work is the image processing library known as woolz. woolz is unusual in that it has separate data structures for the spatial domain of an image object and the image values, which can be bit grey values, rgb α or float/double. the domain of an image object is simply the region of 2-d or 3-d space over which the image is defined. this form of record is particularly appropriate for biomedical images, as it allows a compact representation of images through not storing background values outside of the foreground object of interest and it encodes arbitrary regions of space without the need for image values. internally woolz represents a 2d domain as a number of line segments or intervals for each line for which there is image information. in this way an arbitrary region of 2d space can be defined in a very compact form and binary operations can be reduced to a series of interval comparisons which are very efficient for binary-set, morphological and labelling operations. a 3d domain is simply a planewise stack of 2d domains and null entries for planes which are entirely outside of the domain.each woolz object has its own coordinate system defined with respect to a global origin of all objects and three coordinates kl and p for the columns, lines and planes respectively. the objects have uniform sampling and real world coordinates are achieved through a voxel size which is stored in the domain. because objects can be located at any coordinate location, including negative values, the domain and value table are defined relative to the overall image bounding box which allows for very simple re-location of image data. note conversion from woolz to more standard rectangular domain based image formats may lose this information. converters exist to transform woolz image objects to a wide range of other formats including nifti, vtk, amira and standard tiff-stacks. large woolz objects a recent extension of woolz allows operations on large objects that do not fit in system memory or would take many minutes to load from disk. it uses memory-mapped files, which reside on the external memory and are loaded on-the-fly block by block as the system requires it. 3d memory mapped objects are represented as a domain and grey value pair, however the grey values are stored in fixed size cuboidal blocks, accessed through a lookup table. the image value subdivision into cubes is similar to other solutions but uses memory mapping and file-system block alignment. these block sizes are integer multiples of the file system blocks enabling very efficient read and write operations. with solid-state disks the i/o rate is similar to data that resides on the main memory. cutting arbitrary sections we specify an arbitrary section through the data volume as a plane perpendicular to the line-of-sight. this line is determined by three angles of rotation, two to establish the line direction and the third the orientation around that line. these correspond to pitch, yaw and roll and can be defined in terms of the standard euler angles. the rotation centre is the “fixed point” and the section plane is set by the distance along the view direction. the details of the transform are published, are given in the mapaint technical report and are illustrated in figure it is defined in terms of the fixed point f, angles pitchyaw and rollscale and distance d. the viewing transform can be represented as an affine transform where the section plane is the plane of constant z″ = d. by representing the transform in this fashion is straightforward to include any further re-scaling arising from non-isotropic voxel sizes. these underlying coordinate transformation methods have been extensively used for developing the emouseatlas models and gene-expression database and are part of the essential data captured as part of each submission. convenient navigation with fewer free parameters is provided by a number of standard viewing modes defined in table iip3d server the iip3d server is c++ software, based on a gnu gpl implementation of the iip server by pillay and pitzalis. this is a fastcgi web server module that is managed by the web server or as a standalone application and is connected by the web server as requests arrive. the full technical detail of the woolz extensions of the iip was provided earlier, here we present a limited set of iip3d commands. iip3d request the iip3d server requests are parametrised http requests and typically have four components: server address: identifies the iip3d server. here we use the http protocol and therefore the address is the url of the fcgi application e.g.. resource specifier: the second component of a request defines the image resource from which tiles are requested. this for pyramidal tiffs is, while for 3d woolz objects is the command. view parameters: specifies the parameters of the sectioning plane and the image return as well as setting for overlay colours and opacities. information request: defines the specific information required from the object and sectioning plane set. this can be a tile, a full section or parameters of the object or sectioning plane. the supported image request commands are similar to the iip: returns the full image; while and the jpeg-, png-compressed and uncompressed tiles. iip3d also supports png format with lossless compression and alpha channel, both crucial for painted domain or textual overlays. 3d browsing is essentially the selection of a section by setting the various parameters defined by the,,, and commands. the iip3d protocol also supports the viewing modes from table which given pitch and roll will define yaw. in addition the iip3d server can provide information on the 3d object, with the available queries listed in table these include information about the 3d image volume, coordinate transforms between the displayed tiles and the original 3d image, image value and true voxel sizes. image overlays in a number of atlases, the histological image may have overlay layers presenting for example anatomical domains or gene expression patterns. these are analogous to satellite images overlaid with street map or traffic information. iip3d provides this functionality with the select () command and uses the compound woolz object format to capture the set of domains. this is an array of independent woolz objects stored in a single file and each domain can be accessed by selecting the corresponding index. any number of such indexed domains are allowed and there are no spatial restrictions, the domains can overlap within the context of the viewed section and will be displayed as a set of overlapping colours. the most general syntax of is where is the selected objects index, while,,, are red, blue, green and alpha channels specifiers.the rgb components paint with the chosen colour the selected domain, or for grey/colour objects filter the image selecting only the desired colour components. simplified syntax allow and parameters with; and, with as the default for the omitted parameters. grey level or domain components are either independently selectable or stacked in an arbitrary order to generate grey level images, semi-transparent mask images, or a combination. a sequence of arbitrary number of commands specifies the composition order with the alpha channel information to determine layer visibility. the requests the index or image value at query point allowing augmented display in the viewer application. uniquely the compound object based representation allows multiple overlaying domains to be displayed and queried which implies that the index query will in general return a list. caching faster image delivery is supported by three caching levels. as in the original iip server, the iip3d server caches the requested tiles. if multiple users access the same data, or it returns to a previously visualised region then the tiles are not recomputed but fetched directly from the cache. 3d images are often large, and the disk-read time is a significant overhead. therefore woolz image objects once read are cached for further operations. however, for very large, memory mapped objects only the domain is cached, and not the values. sections transforms which provide a fast look-up-table encoding of the affine transform are also cached. extreme data sizes biomedical images may easily be many gb in size, for example the visible human image is about gb and data sets for electron-microscopy data now reach many terabytes. most current desktop workstations do not have sufficient memory to load such images and are incapable of being upgraded to be able to do so. even when a workstation is capable of reading such a large object into memory the time taken to transfer it from disk is usually inconvenient; with a disk based file system capable of transferring data at 40mb/s it would take at least minutes to read the visible human image into memory. if the image is part of a resource then downloading significant data is very slow and copying multi-terabyte volumes infeasible. in most cases however, only a small subset of the image data is needed for visualisation since the typical user will be restricted by screen size. for this case we have extended the iip3d server to use the memory-mapped option for very large woolz images enabling access to images of tens or hundreds of gigabytes. supported image formats the woolz imaging system can convert from a range of 2d and 3d formats and are therefore directly supported by the iip3d server. supported formats include: amira lattice, stanford density, gif, analyze, iplab, jpeg, biorad confocal, pnm, raw image data, tiff, nifti and vtk. to get the benefit of memory-mapped option for very large volumes and the domain overlays with a global coordinate system, it is necessary to convert to woolz format data. iip3d viewer the iip3d viewer is a configurable web application which displays tiled images requested from a iip3d server. it was initially based on the viewer of pillay but has now been substantially developed and re-organised. the iip3d viewer uses a number of web technologies including: html/http, php/jsp, css, javascript, ajax. a json configuration file identifies various system parameters including the location of the image and the iip3d server. it also defines the parameters of all the layers associated with an image, including name, location of data and display order. if a tree control is to be used, the structure and content of the tree is defined in a configuration file. the central area of the viewer displays the visible region of an image. tools for user interaction with the image will typically also be included. the choice of tools and their respective position within the web page is configurable by the web page developer via another json file. tools, see figure for a selection, are available which allow for: changing the visible region of an image; changing the zoom level of the displayed tiles; toggling the visibility of each layer on/off; changing the opacity of each layer; changing the colour filtering to be applied to grey-level layers; for 3d wlz images changing the sectioning plane. if overlaid data is available and has been defined in the configuration file, it is possible to generate feed-back information by use of a ’mouse-down’ action over the image. this can be used, for example, to identify structure names within an image. with a small amount of javascript coding it is possible for a web-page developer to add their own custom tools. the browser tries to be efficient in terms of transmission bandwidth and client memory by: requesting tiles only for the visible region of an image; requesting tiles only at the current resolution; for a scrolled/panned object, only new tiles are requested; layers which are not visible are not loaded. iip3d proxy in the case where a large image archive it to made accessible via http requests it is often not desirable to have the web-server also act as the image server. in addition with a large archive or high access rates the requests may need to be delegated to a number of servers. to provide a single point of access and enable this delegation to image servers separated from the internet by a firewall, we have developed iip3dproxy, an fcgi application, which filters requests and forwards them to a set of iip3d servers. the communication conforms to the fcgi protocol. though it was designated to work for iip and woolz requests, it is generic and can route any fcgi request, hence it allows chaining of multiple proxies. a multi-iip3d server architecture is shown in figure iip3dproxy is an independent program running on the proxy server. the web server forwards it the fcgi request on a configurable ip port, then the html request string is checked by iip3dproxy and if the definition string of any remote iip3d server is a substring of the request parameters then this query is passed over to the matching server. if no correspondence was found then the request is passed to the default server.  iip3d server tiled image an example section query that consists of six tiles is shown in figure the requests, each returning a png tile, differ only in the ptl command with tile number sweeping to while all have in common a fcgi application, woolz object definition () and sectioning image parameters. overlays with sel the iip3d server delivers tiled views of images and any given application can request tiles from multiple image volumes to be co-displayed as a series of layers with transparency provided by the browser display capabilities. we have implemented a novel strategy using the iip3d command which can be used to request graphical overlays stored in an indexed object. this process is depicted in figure and overlay tiles with anatomy regions defined either individually or collectively can be retrieved and displayed as independent single- or multi-layer, overlay images. the overlay request includes the colour and opacity for requested region. a critical property of the woolz indexed object is that the domains represented are not limited by the number of bits used for the indexed image and and the regions can overlap i.e. they are not spatially exclusive. the woolz iip3d server provides the domain overlay as a single tiled image computed on-the-fly from the woolz object. queries the iip3d protocol includes extension of the query commands to provide information for image layout and user feedback. these include the size of a given section image, the size of a tile, the voxel size, the object volume and bounding box and the grey or colour value at a given pixel location within the section image. for example, the voxel grey value query with respect to the 3d location is http://localhost/fcgi-bin/wlziipsrv.fcgi?∖wlz=/objects/ts wlz∖pab=,190∖obj=wlz-grey-value which results in the reply wlz-grey-value:73 this query can also be made with respect to the 2d location within a given tile. performance evaluation husz et al demonstrated the performance benefit that is realised with a tiled representation with a frame rate of at least fps on a standard wan and fps to a domestic broadband. however, evaluation under heavy load or simulated user interaction had not been studied. here we present the results of testing a single iip3d server with up to sixty simultaneous users. the test consisted of repeatedly running the recorded browsing pattern of a biologist user viewing a cropped visible human data set with a data volume of gb. the browsing pattern included tile requests, down-scaled full section request and object queries, table the total requested data over seconds of interactive browsing is mb, the tile size was × in the following we test two scenarios, first we respect the timing of server requests and second they are ignored which results in a more severe test of the server response. in addition we also test the client requests with and without a random perturbation of the view parameters. this is to ensure all requests are unique and we test the server response without any benefit of caching. a series of experiments were performed with a single iip3d server and to clients. the server and client computers were all equipped with dual 3ghz intel xeon cpus, r.p.m. sata disks and 32gb ram and are directly connected via a dedicated gb ethernet switch. there was no other activity on the server, on the clients or on the network other then operating system management tasks. for testing, we used standalone c++ executables started simultaneously on the client nodes. for test cases with more clients than available client-nodes, multiple concurrent instances were started on the same client node. the timestamp of each request was recorded along with all other iip3d request parameters. this allows replication of user browsing behaviour. the object queried was already prefetched from the disk into the server main memory and the complete set of requests were repeated as whole series and timings captured. the first test set takes s, while the consecutive requires only s, suggesting the effect of the local tile cache. four different scenarios were evaluated with the results shown in figure first, we request in parallel the same data on a number of to clients. the average response time increased from s to s. the time normalised to the number of clients, shows a constant time behaviour, confirming the linearity in figure 7a. the timing requests were ignored in the second test scenario, and requests were started one after the other. the server becomes heavily loaded by the simultaneous requests. however, the result graph shows again a linear behaviour, with ± s per client response time.with its tile caching, the server avoids re-computation of tiles. therefore, to force different section generation, we have have altered the original requests by adding a random value to the sectioning plane angle (). again, figure there is a linear increase of the processing time with the number of requesters. the average per client time is ± s respectively ± s. these times are comparable with the initial s tile cache initialisation for our first test. key times of these results are summarised in table the normalised figure is the average per client. it is interesting to remark in figure 7b that the running time of the timed test sets recorded over seconds, it is respected for the cached scenario, while for the non-cached scenario up to clients managed to fit in the time user needed to browse through the object. this suggests that up to clients may be served without major performance degradation, however as is shown above the display delay increases. client executables were started remotely from a test script. we had no control on the exact timing of these processes neither how the request are prioritised by the apache web server, therefore occasionally “lucky” clients occur with lower execution time then the average. note the client executable did not use any local caching therefore the benefit of the browser cache is not included with these measurements which are therefore a lower bound on the performance experienced by the user. iip3d viewer iip3d is configurable for the application in use. we present the basic 3d browsing functionality, followed by the support of handling overlaid 3d image data and large databases. we exemplify these functionality on the male visiblehuman dataset, adult mouse muti-modality waxholm data, hudsen developing human brain atlas, fly brain atlas and the emouseatlas atlas models. 3d browsing and multi-channel data large volumetric data large objects stored and visualised at high resolution may be read and served by the iip3d server. these use the memory mapped data-structures that avoid the requirement to read the full dataset into main memory which for the larger volumes would not be feasible. in addition for archive data of many 1000s of smaller 3d volumes it enables fast access to any image where again it is not feasible to maintain the image data in memory. we have tested the system for image volumes ranging from to gb and found that the response was slightly slower but acceptable given that these are section views that can not for realised by any other system. in figure we show a view through a composite image comprising full sets of the visible male 3d volume. this results in a 3d image with × × full colour voxels with a data volume of about 138gb. the memory mapped data format also allows us to routinely serve sections views from a multiple 3d image archive which now exceeds 4tb. annotation overlay  we have extended a protocol standard for tile-based viewing of 2d image data to include 3d volumetric data to enable virtual re-sectioning at arbitrary orientations through the volume. the protocol is also extended to handle index-image data giving control to the user of colour and transparency on each individual labelled region. on the basis of this protocol we have implemented an iip3d server to deliver tiles for any requested section and a javascript/ajax viewer which provides a user-interface that will run within a web-browser. performance tests on the server show that it can handle up to simultaneous users without significant loss of performance as experienced by a typical user. the server can be scaled using a series of proxies to provide the required performance for higher access rates as required. the server code is implemented on a standard linux-based machine and does not require special a hpc facility. we have tested this system in the context of atlas-based biological data however it is clearly applicable to a wide range of image data for which arbitrary re-sectioning is a useful visualisation. this visualisation is typically the core requirement in basic science but also medicine where views through the data are the primary views needed by experts who want to see “the real data”. it is complementary to recent development of in html in particular webgl which is exemplified by the google “body browser” demonstrator. we plan to use iip3d in a future webgl based application to provide cut section image textures. for archives of volumetric data or single very-large image volumes this server development is the only plausible mechanism to deliver views to users who do not have the required bandwidth, local storage capability or the compute capability to be able to download and view the data. here we realise the benefit of low-bandwidth tile-based access to image data coupled with very efficient image-processing to calculate the virtual section tiles. this technology is open-source and will run on any unix-based web-server. the iip3d server allows access to independent software either as a standalone application or a more complex java or adobe flash viewer. however, for flexibility and interoperability we provide a javascript api and viewer which runs in most up-to-date internet browser software, e.g. internet explorer, safari, firefox and chrome but with some variation on behaviour arising from different ways in which the browsers have implemented standards. through the iip3d api the 3d viewer can be integrated into complex applications such as next generation embryology - a 3d spatio-temporal framework that augments iip3d data in conjunction with a repository to deliver research and educational material. users may add supplementary augmentation to the 3d volume, together with texts, videos, external links. the compound representation of multiple images and providing them as individual or composed images allows compact representation of related data, and offers a novel 3d visualisation of overlays. also, it has power of representing overlapping domains that standard index-image representations can not provide. we have shown that the performance of our implementation of iip3d scales well with the number of parallel requests. grey level or colour data is successfully augmented with domain or gene expression data. controls, such as sliders, layer selection tool or locator resemble functionality of know tools from desktop applications. these tools and the associated tiling code are all available as open-source. future developments of this software include grey-level transforms such as range slicing for 16bit medical data, extension to vector and scalar data and the introduction of a standard glyph library for location and directional marking. in the context of the client interface we will address usability by undertaking a user-evaluation study. availability and requirements · project name: woolz iip · project home page:http://www.emouseatlas.org/emap/analysis_tools_resources/software/wlziip.html · operating system: sever: linux; client: platform independent · programming language: server: c and c++; client: javascript · other requirements: server: woolz x.x, fastcgi library and web server with fastcgi enabled; client: internet browser must support javascript and display images · license: gnu gpl · any restrictions to use by non-academics: according to gnu gpl endnotes ahttp://www.emouseatlas.org/emap/help/compatibility.htmlbhttp://research.nesc.ac.uk/node/400 abbreviations iip: internet imaging protocol; iip3d: 3d extended internet imaging protocol; fcgi/fastcgi: fast common gateway interface. competing interests the authors declare that they have no competing interests. author’s contributions zh designed and extended the original iip protocol, coded the iip3d server, suggested the existing compound woolz object representation for domain data and wrote the first draft of this paper. bh and rb develop the woolz image processing library, rb developed the woolz sectioning algorithm and bh added support for memory mapped objects. nb integrated features of existing browser to provide a unified application programmer interface. nm added support of tree selection of anatomical hierarchy for overlaid images and provide input for viewer api standardisation. rb formulated the underlying ideas for using the iip server for section data and managed the design and software development. all authors read and approved the final manuscript. 
interactive browsing of arbitrary sections through large biomedical-image volumes is made possible by use of an extended internet protocol and efficient server-based image tiling. this is aggravated when users need to access tens or hundreds of such images from an archive. the tools open the possibility of enabling fast access to large image archives without the requirement of whole image download and client computers with very large memory configurations. the standard internet imaging protocol has been extended to enable arbitrary 2d sectioning of 3d data as well a multi-layered images and indexed overlays. here we solve the problem for 2d section views through archive data delivering compressed tiled images enabling users to browse through very-large volume data in the context of a standard web-browser. we have tested the server software on a low-cost linux-based server for image volumes up to 135gb and simultaneous users. the extended protocol is termed iip3d and we have implemented a matching server to deliver the protocol and a series of ajax/javascript client codes that will run in an internet browser.
 large-scale volumetric biomedical image data of three or more dimensions are a significant challenge for distributed browsing and visualisation. many images now exceed 10gb which for most users is too large to handle in terms of computer ram and network bandwidth. this is aggravated when users need to access tens or hundreds of such images from an archive. here we solve the problem for 2d section views through archive data delivering compressed tiled images enabling users to browse through very-large volume data in the context of a standard web-browser. the system provides an interactive visualisation for grey-level and colour 3d images including multiple image layers and spatial-data overlay.  the standard internet imaging protocol has been extended to enable arbitrary 2d sectioning of 3d data as well a multi-layered images and indexed overlays. the extended protocol is termed iip3d and we have implemented a matching server to deliver the protocol and a series of ajax/javascript client codes that will run in an internet browser. we have tested the server software on a low-cost linux-based server for image volumes up to 135gb and simultaneous users. the section views are delivered with response times independent of scale and orientation. the exemplar client provided multi-layer image views with user-controlled colour-filtering and overlays.  interactive browsing of arbitrary sections through large biomedical-image volumes is made possible by use of an extended internet protocol and efficient server-based image tiling. the tools open the possibility of enabling fast access to large image archives without the requirement of whole image download and client computers with very large memory configurations. the system was demonstrated using a range of medical and biomedical image data extending up to 135gb for a single image volume. 