BACKGROUND
biomedical image processing is fundamental to many biological research methods  <cit> .
these algorithms take parameter values and images as input, and produce annotated images and quantitative measures as output.
because they are sensitive to parameter values, imaging artefacts, and factors like tissue type, it is difficult to find robust parameter values that ensure high-quality output.
consequently, user judgment is an integral part of the optimisation process.
optimisation problems may be classified in different ways, including the scale of parameter and output space.
for the class of problem we consider, users deal with 2- <dig> input parameters and 2- <dig> output measures.
users also want to review image-based output for up to five images.
we obtained these numbers by consulting domain experts and by observing users.
they also correspond to observations in previous work  <cit> .
there are problem classes with more parameters, but they are beyond the scope of this paper.
in this section we first review existing approaches for parameter optimisation.
we then identify two important challenges  and show that they are not addressed by this work.
in further sections we describe a novel visualisation method to address the challenges and discuss a case study where our approach was used.
visual parameter optimisation
the most prominent approach for parameter optimisation is parameter tweaking.
this involves repeatedly adjusting parameter values and reviewing output.
it is tedious and incurs time and quality costs  <cit> .
automated parameter optimisation methods also exist, but require specialised mathematical insight and do not allow subjective analysis of output  <cit> .
to address the shortcomings of parameter tweaking and automation, a number of visualisation methods have been developed .
we classify them as follows.
first, guided navigation approaches rely on an objective function or a distance measure from an ideal output .
some show neighbourhoods in parameter space to guide users to optimal values  <cit> .
others support systematic exploration of parameter space by combining modelling, simulation, and visualisation  <cit> .
these methods require an understanding of complex mathematical concepts for interpretation, which users may find challenging.
also, objective functions and ground truths are not always available .
a second class of methods relies on interactive visual exploration and qualitative evaluation of output.
this includes dynamic queries of distribution plots of input and output  <cit> .
it is also possible to visualize the parameter search graph to let users revisit and refine existing outputs  <cit> .
other methods visually structure parameter space to support the identification of suitable values  <cit> .
an alternative is to emphasise the characteristics of output space and to let users select the output that best suit their needs  <cit> .
third, parameter space is typically high-dimensional and standard multidimensional visualisation methods can be used.
this includes: dimensional stacking, where data items are embedded in a hierarchy of nested scatterplots  <cit> ; hierarchical clustering, similar to dimensional stacking, but nested dimensions are shown as a directed tree; scatterplot matrices, where a matrix of scatterplots shows all two-way combinations of dimensions  <cit> ; and parallel coordinates, where every variable is represented by a parallel axis and data items by polylines that intersect the axes.
hierarchical clustering and parallel coordinates have been extended to embed one image per parameter combination  <cit> .
finally, there are methods specifically for considering parameters in conjunction with image-based output.
"open-box" methods are custom-developed for specific image processing algorithms  <cit> .
as parameter values are changed, they show algorithm-specific intermediate measures and update an output image.
in previous work, we presented a visualisation method to analyse input parameters and image-based output for arbitrary algorithms  <cit> .
it shows a cartesian sampling of parameter space as a tree with a node-link diagram.
users select smaller contiguous regions of sampled parameter space to view associated output images in a coordinated view.
this was followed by a case study, where users were able to analyse larger parts of parameter space and achieved higher quality results compared to parameter tweaking  <cit> .
challenges
we now describe two unaddressed challenges, identified by analysing the above case study, a review of related work, and discussions with domain experts .
we omit guided navigation since objective functions and ground truths usually do not exist in our application domain.
multiple inputs and outputs
image processing algorithms require multiple input parameters to be set and users are usually interested in analysing the results of an algorithm on multiple input images .
users also want to examine the multiple output images and multiple output measures generated during algorithm execution.
hence, users need to combine an objective analysis of parameters and measures with a subjective analysis of images.
in table  <dig>  we show the ways in which previous visualisation methods are deficient.
all methods deal with multiple input parameters, though some only treat pairs  <cit> .
however, none support the analysis of algorithms applied to multiple input images.
further, previous methods are not designed to support visual analysis of multiple output measures .
standard multidimensional visualisation methods can visualise parameters and measures but generally do not cater for images .
in sum, there is an unmet challenge to visually support analysis of multiple input parameters, input images, output measures, and output images.
understanding
helping users understand their image processing algorithms is an important requirement to achieve confidence and generalise findings.
with confidence in optimal parameter values for input images of the same class , users can automate the processing for large volumes of similar data.
an understanding of underlying algorithms also lets users generalise their findings to process input images with different characteristics.
finally, there is a need to validate image processing algorithms and to identify errors, particularly in a research context.
in table  <dig>  we show that previous visualisation techniques provide incomplete support for understanding algorithms.
a few were designed to support understanding of specific algorithms, but do not generalise beyond that  <cit> .
there are also methods that emphasise relationships between input and output.
for example, our previous work helped users discover implementation errors and a logic error in a segmentation algorithm  <cit> , while another allows for relating similar or erroneous output to parameter values  <cit> .
some methods let users investigate different scenarios in terms of input parameters  <cit> , support the parameter search process  <cit> , or permit exploration of simulations in a goal-oriented manner  <cit> .
nonetheless, they are geared to finding suitable parameter values.
we conclude that the challenge of supporting understanding is unmet.
for previous visualisation methods, understanding is a bonus and not a design objective.
methods
to address the above challenges, we developed a visualisation technique to optimise parameters for biomedical image processing algorithms and implemented it in a tool called paramorama <dig>  <cit> .
our technique is novel because: it enables holistic analysis of numerical and image-based inputs and outputs, and it provides interactive capabilities to enable flexible exploration of relationships between inputs and outputs.
this work is the result of  <dig> months of close collaboration between the authors and diverse domain experts.
although our approach extends our previous work  <cit> , there are important differences in conceptual approach, visual design, and the analysis it supports.
in this section, we describe our design decisions.
our data is generated offline by taking a cartesian sampling of parameter space.
for each  input parameter, a user-specified interval is sampled.
for each input image, the algorithm is executed once for each unique combination of sampled parameter values.
this generates multiple output measures and output images that are associated with a particular combination of input parameters and the set of input images.
output measures are descriptive metrics that capture information about the output.
we refer to a unique combination of input parameters, input images, output measures, and output images as a data record.
multiple inputs and outputs
as shown below, we combine a tabular visualisation of input parameters and output measures with an image browser for input and output images.
we also describe design alternatives that we considered.
tabular visualisation
we show the relationships between input parameters and output measures in a tabular visualisation ).
columns at the left represent parameters and columns at the right represent measures.
each data record is represented by a row that spans across the columns.
the value taken for a parameter or measure is encoded in the corresponding column.
if the vertical space per row is more than four pixels, a bar chart encodes every column, otherwise a line chart is used.
although line charts do not prevent over-plotting, they are effective to let users discern high-level patterns when limited vertical space is available.
our tables are similar to a table lens, which was developed primarily as a focus+context method  <cit> .
our objective, however, is to use the tabular representation to assist users in flexibly identifying and analysing relationships between parameters and measures.
as we will show, we achieve this by extending our method with a number of interactive features.
image browser
our method has an image browser at the right of the user interface.
it shows a horizontal list of input images at the top ).
when users select rows in the tabular visualisation, the corresponding output images are shown below the input images in a grid ).
column i shows the output images produced by applying the algorithm to the ith input image.
each row represents a data record and the top-to-bottom order corresponds to the order of selected records in the tabular visualisation ).
by viewing the column of output images below each input image, users can compare output produced by different input parameter combinations for different input images.
each output image is blended with the input image to make comparisons easier .
users can also define a rectangular region of interest in each input image to view for the output images.
this helps when there are particular regions that are known to be problematic for an algorithm  and 1).
users can also adjust output image magnification.
the primary mechanism to analyse relationships between input and output is interaction .
as an additional aid, we provide a summary where horizontal strips represent the domains of parameters and measures ).
dark regions indicate the values to which currently displayed images correspond.
design alternatives
we also considered alternative visualisation methods.
the most applicable highlight the structure of input or output space, but no existing ones integrate both .
standard multidimensional visualisation methods were ruled out for the reasons below.
for dimensional stacking and hierarchical clustering, the real-estate requirements increase exponentially with the number of dimensions.
scatterplot matrices can visualise an arbitrary number of dimensions but, due to perceptual limitations, it is difficult to analyse relationships that span across more than two.
for example, the multiway correlations that show up as nested patterns in figure  <dig> cannot be easily discerned in figure  <dig>  which shows the same data.
our approach directly shows cyclical patterns in columns .
by contrast, parallel coordinates often mask such patterns when polylines overlay each other, requiring further interaction ).
to highlight cyclical patterns, we also considered spiral representations .
these require tuning an additional parameter to find a rotation interval and do not support multidimensional data.
in fact, these alternatives all require far more effort for interacting with the data than our approach .
while developing our image browser we considered existing work for browsing photo libraries.
some, like photofinder  <cit> , show grids of sequentially ordered images.
others, like photomesa  <cit> , show the hard disk directory structure as a treemap.
these methods were not designed to show relationships with and facilitate understudying of associated inputs and outputs.
understanding
users need to discover and analyse relationships between input and output.
interaction is key to flexibly select data records and inspect associated images.
for this, we combine column-based sorting, including automated sorting, with context-sensitive selection.
column-based sorting
users can interactively sort the rows in the tabular visualisation to identify relationships, such as correlations, that span across input parameter and output measure space.
it is precisely this type of analysis that previous methods, including our own work, do not enable users to perform, instead focussing either on input or output .
when users click on a column header, data records are sorted by the values of the corresponding input parameter or output measure.
in figure  <dig>  the data have been sorted by the second parameter .
a step-like pattern has emerged where records are grouped into a number of bins with the same value for parameter p <dig>  it is possible to identify relationships between p <dig> and some of the output measures at the right.
when multiple columns are selected, the order in which they were selected matters and all previously applied sortings are maintained.
figure  <dig> shows the result of sorting figure  <dig> on p <dig>  the records are only reordered within each of the bins of p <dig> to show a nested step-like pattern.
now, even more striking relationships with the output measures appear.
automated sorting
during prototyping, we repeatedly observed users searching for the parameter that most highly correlates with output measures.
we consequently implemented a simple automated sorting facility that we call "smart sorting".
when users click on "smart sort" , our method computes the aggregate correlation of each input parameter and all output measures.
the parameter with the highest correlation is identified and the data records are sorted by this parameter ).
context-sensitive selection
during prototyping, users found selection of individual data records too tedious to effectively analyse relationships between input and output.
to address this, we developed context-sensitive selection.
suppose the cursor intersects row r and column c. in addition to highlighting row r, all directly adjacent rows with the same value for column c also receive focus.
for example, compare the highlighted rows in figure  <dig>  where the cursor intersects column one, to figure  <dig> where it intersects column two.
rows in focus are enclosed by a red frame and marked by two red disks in the margins.
clicking selects all rows in focus and marks each selected record with blue disks.
compound selections are made by multiple selections of this type.
clicking on the button labelled "show", below the tabular visualisation at the right, displays all images associated with selected data records in the image browser ).
to provide more flexibility, users can rapidly filter records by clicking on "filter" below any column and then specify an interval of interest.
all records where the corresponding parameter or measure falls outside the interval are hidden.
a strip below each column indicates which parts of its domain are currently displayed .
there are situations where users want to look at output images associated with a single data record in real time.
by holding the shift key, output images for the table row directly under the cursor are temporarily shown in the image browser.
images for a single record can be read and drawn at interactive speeds.
design alternatives
we also investigated updating the image browser in real time as users select data records, or to use image caching.
the former imposes a performance penalty for reading large numbers of images from disk, while the memory footprint of the latter limits scalability.
column sorting combined with context-sensitive selection is an effective and efficient way to investigate meaningful subsets of data.
we also considered "hard sorting" rows by column  <dig>  then by column  <dig>  and so on.
this imposes a column-based hierarchy on the data and forces users to reorder columns to change the hierarchy.
instead, our approach lets users choose a column to sort on with one button click.
for automated sorting, it is possible to rank all input parameters on their individual correlations and to sort the data by all parameters, in this rank order.
however, our users indicated that they prefer sorting by the single most significant parameter and our method therefore implements this approach.
providing automation as a "one-click" option, which can be visually verified and easily undone, alleviated fears about added complexity introduced by automated analysis.
RESULTS
in this section, we describe applications of our method by providing an intuitive example and a case study.
we also consider further biomedical applications, lessons learned, and opportunities for future work.
example: cell nuclei detection
cell nuclei detection is an important but challenging step in many high-throughput assays.
as demonstrated in additional file s <dig>  figure  <dig> shows the results of a cell nuclei detection algorithm for photomicrographs of human ht <dig> cells  that had been stained  to highlight nuclei  <cit> .
the algorithm has five input parameters that had been sampled three times each on meaningful intervals.
for each combination of parameter values, the algorithm was run on two input images.
for each parameter value combination, object counts for each image were captured as output measures and the outlines of detected nuclei were saved as output images.
the results have been sorted on column  <dig>  the nuclei count for the first input image.
this shows relationships with the second  and fourth  parameters.
by considering the values of the two parameters in combination with the output measures and images, it is straightforward to identify values for both that produce accurate nuclei detection .
these results have been selected in figure  <dig>  a biomedical novice  was able to identify these values in less than five minutes.
the same task takes more than an hour with conventional parameter tweaking  <cit> .
case study: colour deconvolution for histology
histology is the study of tissue at a microscopic scale.
tissue is sectioned into micrometre-thin slices, fixed to glass slides, and stained with dyes that highlight different cellular compartments and structures.
hematoxylin and eosin dyes are often combined  to, respectively, colour cell nuclei blue and cytoplasm and connective tissue pink ).
however, different sub-cellular structures and proteins have overlapping absorption spectra, which makes it hard to differentiate contributions of individual dyes.
colour deconvolution is an image processing method that can extract individual dyes.
this has important biological implications because it allows for quantitative tissue characterisation .
for colour deconvolution, a suitable deconvolution matrix must be found.
when applied to the original rgb image, this matrix splits it into several images, each representing the contribution of an individual dye.
a biomedical image analysis expert  had been investigating the assumed optimality of ruifrok and johnson's deconvolution matrix  <cit> .
he had two objectives.
develop quality measures to quantify the performance of colour deconvolution.
this requires an understanding of the behaviour of the deconvolution method.
find optimal values for two corrective parameters to optimise the deconvolution matrix.
if non-zero, this would show that ruifrok and johnson's method is not optimal in all cases.
the expert had been working on the problem just short of a year when we got involved.
the description below of how he achieved o <dig> and o <dig> with our method was obtained by a diary study and follow-up interview.
preparation
the expert worked on two tissue types, liver and lymphoma , which had been stained with h&e  <cit> .
by using prior knowledge of the light absorption properties of tissue , the expert had developed two corrective parameters, p <dig> and p <dig> to apply to the deconvolution matrix and six candidate output measures to quantify the output quality.
the first parameter  characterizes the haematoxylin stain in absorbing the red, green and blue components of the incident light.
this parameter can be used to isolate cell nuclei in most cases.
the second parameter  characterises the eosin stain and can be used to isolate cytoplasm and connective tissue.
respectively, m1-m <dig> are the percentage of negative coefficients, mean value of negative coefficients, and standard deviation of negative coefficients corresponding to the first stain, while m4-m <dig> record the same results for the second stain.
detailed rationale of these parameters and measures are beyond the scope of this paper , but an appreciation of the value of the insights gained with our method does not rely on a specialist understanding.
the expert was keen to find an alternative to parameter tweaking and immediately saw the potential of our method.
he customised his software to sample the input parameters  <dig> times each on intervals identified based on domain knowledge.
this yielded  <dig> unique combinations.
next, for each of the two input images, colour deconvolution was performed for each unique combination of sampled parameter values, and the corresponding data record was saved to disk.
input and output images measured  <dig>   <dig> ×  <dig>   <dig> pixels.
visual analysis
the expert's data is typical for our application domain.
each data record contained two input parameters, two input images,  <dig> output measures , and four output images .
the expert started by analysing the deconvolution results for the liver section image ).
o <dig>  after loading the data, the expert applied smart sorting.
this identified and sorted the data on p <dig>  the parameter with the highest aggregate correlation with the variance of the output measures ).
the expert observed a number of patterns suggesting correlations between p <dig> and several of the output measures.
some appeared to be direct, while others appeared to be inverse relationships.
the expert then sorted on p <dig>  he was surprised by the result shown in figure  <dig>  where clear correlations between the input parameters and the output measures are evident.
before using our method, he had only seen single results  and had not been able to piece together the relationships shown here.
the expert now analysed these relationships ).
first, as m <dig> and m <dig> only took one of two values, they did not appear to carry sufficient information and the expert discarded them, reducing the complexity of the subsequent investigation.
next, the expert noticed that m <dig> and m <dig> appeared to be inversions of each other: when m <dig> increases, m <dig> decreases and vice versa.
finally, output measures m <dig> and m <dig> also closely resemble each other.
the expert concluded that m <dig> and m <dig> are sufficient to analyse the quality of results.
the overview and analysis provided by our method enabled him to understand the behaviour shown by the output measures and, consequently, he was confident in this selection.
he confirmed his choice by performing a similar analysis of the results of colour deconvolution on the lymphoma image.
figure  <dig> shows our method with the results of deconvolution applied to both input images.
the two input parameters p <dig> and p <dig> are at the left of the tabular visualisation.
at the right is the reduced set of output measures, m <dig> and m <dig> for the liver images and m1− <dig> and m1− <dig> for lymphoma.
in this way, the expert was able to address o <dig>
o <dig>  the results in figure  <dig> show a direct relationship between the value assumed for parameter p <dig> and output measures m <dig> and m1− <dig>  based on the assumed optimality of ruifrok and johnson's deconvolution matrix  <cit> , the expert expected to find high-quality output when m <dig> and m1− <dig> approach zero.
also, the parameters p <dig> and p <dig> represent deviations from the original deconvolution row vectors, where their 6th value represents no change.
since the default deconvolution matrix was derived from numerous empirical experiments, the expert wanted to veer away from it as little as possible to achieve improved quality output.
having established p <dig> as the parameter most closely correlated with variation of the output measures, the expert decided to first review those data records closest to where p <dig> takes its 6th value and where m <dig> and m1- <dig> approach zero.
figure  <dig> shows how he selected these records using context-sensitive selection.
the expert next reviewed the output images in the image browser.
because deconvolution splits each input image into two output images , there are four columns of images in figure  <dig>  the first two columns correspond to the output for the liver section input image while the last two correspond to lymphoma.
by reviewing these images, and cross-referencing parameters and measures, the expert identified a combination of input parameter values where p <dig> ≠  <dig> that yielded higher quality results for both input images than the original deconvolution matrix proposed in  <cit> .
this enabled the expert to address o <dig> and show that the ruifrok and johnson deconvolution matrix is not always optimal.
reflection
we followed up by conducting an unstructured debriefing interview.
from this and our analysis of the case study, we conclude the following.
first, with our approach the expert was able to effectively and efficiently address his research objectives.
in particular, by addressing o <dig>  our technique led him to a breakthrough in understanding.
by discovering that an underlying assumption about the deconvolution algorithm he considered is invalid, he showed that ruifrok and johnson's deconvolution matrix is not optimal for all cases.
figure  <dig> illustrates this for the liver section.
second, the total analysis time for both data sets was roughly  <dig> minutes.
in contrast, the expert estimated that an attempt at a similar analysis using his conventional approach  would have required several days.
third, the expert noted that despite previously focusing on o <dig> and o <dig> for nearly a year, he had little confidence in the results obtained with his conventional methods.
by contrast, he was very confident of the results achieved with our method.
in fact, based on his experience, he held a strong conviction that the rigour of analysis that our technique supports is practically unfeasible using his conventional approach.
applications
by applying the above algorithm to stained histology sections of engineered articular cartilage, scientists at the university of leeds have found a direct correlation between stain intensity, which isolates an extracellular matrix, and the compressive strength of the cartilage.
cartilage repair with engineered tissue is an important new regenerative therapy for ageing populations and this approach offers a novel method for quantifying a key property using histology sections that are already routinely taken for subjective inspection.
research into regenerative treatment also requires accurate models of, for example, spinal biomechanics.
for this, biologists at leeds are using the above algorithm to investigate if the distribution of stains obtained by deconvolution can be used to derive models of the structural properties of intervertebral disks.
finally, image processing is a fundamental part of high-content screening workflows.
effective and efficient optimisation of these algorithms dramatically reduces the associated time and quality costs .
lessons learned
the parameter visualisation method described in this paper treats multiple input parameters, input images, output measures, and output images as first-class citizens for the first time.
it results from an evolution in our understanding of the problem space.
this is mirrored in the progression from our initial work that focuses only on input parameters and output for a single input image  <cit> , to a limited and makeshift treatment of measures  <cit> , to the work presented here.
the long-term collaboration between us  and diverse domain experts  has been absolutely essential for this.
during this time, our collaborators' understanding and expectations of the role of visualisation also changed.
our joint work has convinced them that interactive visualisation is an important analysis paradigm.
as our case study shows, visualisation enables them to address their problems in new and more effective ways.
by focusing on the problem, and not intrinsically on technical novelty, we were able to achieve a step-change for users.
due to the gap in previous parameter visualisation approaches, which are either parameter- or output-centric, they were limited in the types of analysis they could perform.
although our approach is partly based on existing methods, it combines these in a novel way to bridge this gap.
by documenting the problem space and the design space, we argue that others will also be able to benefit from this work.
this echoes calls for design studies by other authors  <cit> .
future work
as we show in our case study, our approach increases effectiveness, efficiency and confidence in our application domain, where it is currently very challenging to analyse and understand relationships between multiple inputs and multiple outputs.
our approach also has limitations, however.
the tabular visualisation was not designed to deal with over-plotting and, in practice, is limited to a maximum of about  <dig> parameters and a few thousand unique parameter value combinations.
this deals with a class of problem that our target users typically face, but will not address all applications of parameter optimisation.
for example, population models can contain hundreds of parameters  <cit> .
despite over-plotting, we have successfully analysed just over  <dig>  data records of browsing behaviour from an unrelated usability study, where a total of nine parameters and measures were investigated and where sensitivity plots were viewed in the image browser.
this suggests that our approach could scale to larger data sets than designed for and that it has potential for problems outside biomedical image processing.
still, our approach requires sample sizes to be chosen in accordance with the number of data records to visualise.
we see two ways to cater for scenarios that require greater scalability.
first, a brute-force approach is to visualise more samples by using larger displays such as powerwalls, by letting visualisations scroll, or by implementing focus + context techniques .
a second approach is computational steering, where visualisation is integrated into a larger iterative cycle aimed at specifying and resampling regions of parameter space on the fly .
we can, for instance, envision our visualisation interface integrated into an image processing framework like cellprofiler  <cit> .
more work is required to investigate these possible approaches.
another challenge is scaling to very large numbers of input and output images.
discussions with experts revealed that they would like to follow up on analyses like our case study with larger-scale validation exercises that involve hundreds or thousands of input images and their corresponding output.
this would be valuable to validate the robustness of a set of parameter values.
here the emphasis shifts from dealing with the complexity of parameter space to also dealing with the complexity of very large collections of image-based input and output.
there are currently no methods that enable users to interactively analyse the output generated for very large numbers of input images and this is an important open challenge for future research.
CONCLUSIONS
we presented a visualisation technique for parameter optimisation of biomedical image processing algorithms.
it addresses two challenges: dealing with multiple inputs and outputs  and enabling understanding of underlying algorithms.
to show this, we provided a case study where a biomedical image processing expert used our method for colour deconvolution of histology images.
competing interests
the authors declare that they have no competing interests.
authors' contributions
ajp carried out visualisation design and implementation, conducted parts of the case study, and drafted the manuscript.
yz provided expert input for visualisation design, carried out and helped to draft the case study.
rar helped with visualisation design and to draft the manuscript.
all authors read and approved the final manuscript.
supplementary material
additional file 1
click here for file
acknowledgements
the authors thank anne carpenter and mark bray , duane carey and scott finlay , and tatiana von landesberger  for crucial input.
this research was supported by welmec, a centre of excellence in medical engineering funded by the wellcome trust and epsrc .
write up was made possible by a leverhulme early career fellowship award to ajp .
declarations
funding for publication of this article was provided by welmec, a centre of excellence in medical engineering funded by the wellcome trust and epsrc .
this article has been published as part of bmc bioinformatics volume  <dig> supplement  <dig>  2015: proceedings of the 5th symposium on biological data visualization: part  <dig>  the full contents of the supplement are available online at http://www.biomedcentral.com/bmcbioinformatics/supplements/16/s11

 biomedical image processing is fundamental to many biological research methods. these algorithms take parameter values and images as input, and produce annotated images and quantitative measures as output. because they are sensitive to parameter values, imaging artefacts, and factors like tissue type, it is difficult to find robust parameter values that ensure high-quality output. consequently, user judgment is an integral part of the optimisation process. optimisation problems may be classified in different ways, including the scale of parameter and output space. for the class of problem we consider, users deal with 2- input parameters and 2- output measures. users also want to review image-based output for up to five images. we obtained these numbers by consulting domain experts and by observing users. they also correspond to observations in previous work. there are problem classes with more parameters, but they are beyond the scope of this paper. in this section we first review existing approaches for parameter optimisation. we then identify two important challenges and show that they are not addressed by this work. in further sections we describe a novel visualisation method to address the challenges and discuss a case study where our approach was used. visual parameter optimisation the most prominent approach for parameter optimisation is parameter tweaking. this involves repeatedly adjusting parameter values and reviewing output. it is tedious and incurs time and quality costs. automated parameter optimisation methods also exist, but require specialised mathematical insight and do not allow subjective analysis of output. to address the shortcomings of parameter tweaking and automation, a number of visualisation methods have been developed. we classify them as follows. first, guided navigation approaches rely on an objective function or a distance measure from an ideal output. some show neighbourhoods in parameter space to guide users to optimal values. others support systematic exploration of parameter space by combining modelling, simulation, and visualisation. these methods require an understanding of complex mathematical concepts for interpretation, which users may find challenging. also, objective functions and ground truths are not always available. a second class of methods relies on interactive visual exploration and qualitative evaluation of output. this includes dynamic queries of distribution plots of input and output. it is also possible to visualize the parameter search graph to let users revisit and refine existing outputs. other methods visually structure parameter space to support the identification of suitable values. an alternative is to emphasise the characteristics of output space and to let users select the output that best suit their needs. third, parameter space is typically high-dimensional and standard multidimensional visualisation methods can be used. this includes: dimensional stacking, where data items are embedded in a hierarchy of nested scatterplots; hierarchical clustering, similar to dimensional stacking, but nested dimensions are shown as a directed tree; scatterplot matrices, where a matrix of scatterplots shows all two-way combinations of dimensions; and parallel coordinates, where every variable is represented by a parallel axis and data items by polylines that intersect the axes. hierarchical clustering and parallel coordinates have been extended to embed one image per parameter combination. finally, there are methods specifically for considering parameters in conjunction with image-based output. "open-box" methods are custom-developed for specific image processing algorithms. as parameter values are changed, they show algorithm-specific intermediate measures and update an output image. in previous work, we presented a visualisation method to analyse input parameters and image-based output for arbitrary algorithms. it shows a cartesian sampling of parameter space as a tree with a node-link diagram. users select smaller contiguous regions of sampled parameter space to view associated output images in a coordinated view. this was followed by a case study, where users were able to analyse larger parts of parameter space and achieved higher quality results compared to parameter tweaking. challenges we now describe two unaddressed challenges, identified by analysing the above case study, a review of related work, and discussions with domain experts. we omit guided navigation since objective functions and ground truths usually do not exist in our application domain. multiple inputs and outputs image processing algorithms require multiple input parameters to be set and users are usually interested in analysing the results of an algorithm on multiple input images. users also want to examine the multiple output images and multiple output measures generated during algorithm execution. hence, users need to combine an objective analysis of parameters and measures with a subjective analysis of images. in table we show the ways in which previous visualisation methods are deficient. all methods deal with multiple input parameters, though some only treat pairs. however, none support the analysis of algorithms applied to multiple input images. further, previous methods are not designed to support visual analysis of multiple output measures. standard multidimensional visualisation methods can visualise parameters and measures but generally do not cater for images. in sum, there is an unmet challenge to visually support analysis of multiple input parameters, input images, output measures, and output images. understanding helping users understand their image processing algorithms is an important requirement to achieve confidence and generalise findings. with confidence in optimal parameter values for input images of the same class, users can automate the processing for large volumes of similar data. an understanding of underlying algorithms also lets users generalise their findings to process input images with different characteristics. finally, there is a need to validate image processing algorithms and to identify errors, particularly in a research context. in table we show that previous visualisation techniques provide incomplete support for understanding algorithms. a few were designed to support understanding of specific algorithms, but do not generalise beyond that. there are also methods that emphasise relationships between input and output. for example, our previous work helped users discover implementation errors and a logic error in a segmentation algorithm, while another allows for relating similar or erroneous output to parameter values. some methods let users investigate different scenarios in terms of input parameters, support the parameter search process, or permit exploration of simulations in a goal-oriented manner. nonetheless, they are geared to finding suitable parameter values. we conclude that the challenge of supporting understanding is unmet. for previous visualisation methods, understanding is a bonus and not a design objective. methods to address the above challenges, we developed a visualisation technique to optimise parameters for biomedical image processing algorithms and implemented it in a tool called paramorama. our technique is novel because: it enables holistic analysis of numerical and image-based inputs and outputs, and it provides interactive capabilities to enable flexible exploration of relationships between inputs and outputs. this work is the result of months of close collaboration between the authors and diverse domain experts. although our approach extends our previous work, there are important differences in conceptual approach, visual design, and the analysis it supports. in this section, we describe our design decisions. our data is generated offline by taking a cartesian sampling of parameter space. for each input parameter, a user-specified interval is sampled. for each input image, the algorithm is executed once for each unique combination of sampled parameter values. this generates multiple output measures and output images that are associated with a particular combination of input parameters and the set of input images. output measures are descriptive metrics that capture information about the output. we refer to a unique combination of input parameters, input images, output measures, and output images as a data record. multiple inputs and outputs as shown below, we combine a tabular visualisation of input parameters and output measures with an image browser for input and output images. we also describe design alternatives that we considered. tabular visualisation we show the relationships between input parameters and output measures in a tabular visualisation ). columns at the left represent parameters and columns at the right represent measures. each data record is represented by a row that spans across the columns. the value taken for a parameter or measure is encoded in the corresponding column. if the vertical space per row is more than four pixels, a bar chart encodes every column, otherwise a line chart is used. although line charts do not prevent over-plotting, they are effective to let users discern high-level patterns when limited vertical space is available. our tables are similar to a table lens, which was developed primarily as a focus+context method. our objective, however, is to use the tabular representation to assist users in flexibly identifying and analysing relationships between parameters and measures. as we will show, we achieve this by extending our method with a number of interactive features. image browser our method has an image browser at the right of the user interface. it shows a horizontal list of input images at the top ). when users select rows in the tabular visualisation, the corresponding output images are shown below the input images in a grid ). column i shows the output images produced by applying the algorithm to the ith input image. each row represents a data record and the top-to-bottom order corresponds to the order of selected records in the tabular visualisation ). by viewing the column of output images below each input image, users can compare output produced by different input parameter combinations for different input images. each output image is blended with the input image to make comparisons easier. users can also define a rectangular region of interest in each input image to view for the output images. this helps when there are particular regions that are known to be problematic for an algorithm and 1). users can also adjust output image magnification. the primary mechanism to analyse relationships between input and output is interaction. as an additional aid, we provide a summary where horizontal strips represent the domains of parameters and measures ). dark regions indicate the values to which currently displayed images correspond. design alternatives we also considered alternative visualisation methods. the most applicable highlight the structure of input or output space, but no existing ones integrate both. standard multidimensional visualisation methods were ruled out for the reasons below. for dimensional stacking and hierarchical clustering, the real-estate requirements increase exponentially with the number of dimensions. scatterplot matrices can visualise an arbitrary number of dimensions but, due to perceptual limitations, it is difficult to analyse relationships that span across more than two. for example, the multiway correlations that show up as nested patterns in figure cannot be easily discerned in figure which shows the same data. our approach directly shows cyclical patterns in columns. by contrast, parallel coordinates often mask such patterns when polylines overlay each other, requiring further interaction ). to highlight cyclical patterns, we also considered spiral representations. these require tuning an additional parameter to find a rotation interval and do not support multidimensional data. in fact, these alternatives all require far more effort for interacting with the data than our approach. while developing our image browser we considered existing work for browsing photo libraries. some, like photofinder, show grids of sequentially ordered images. others, like photomesa, show the hard disk directory structure as a treemap. these methods were not designed to show relationships with and facilitate understudying of associated inputs and outputs. understanding users need to discover and analyse relationships between input and output. interaction is key to flexibly select data records and inspect associated images. for this, we combine column-based sorting, including automated sorting, with context-sensitive selection. column-based sorting users can interactively sort the rows in the tabular visualisation to identify relationships, such as correlations, that span across input parameter and output measure space. it is precisely this type of analysis that previous methods, including our own work, do not enable users to perform, instead focussing either on input or output. when users click on a column header, data records are sorted by the values of the corresponding input parameter or output measure. in figure the data have been sorted by the second parameter. a step-like pattern has emerged where records are grouped into a number of bins with the same value for parameter p it is possible to identify relationships between p and some of the output measures at the right. when multiple columns are selected, the order in which they were selected matters and all previously applied sortings are maintained. figure shows the result of sorting figure on p the records are only reordered within each of the bins of p to show a nested step-like pattern. now, even more striking relationships with the output measures appear. automated sorting during prototyping, we repeatedly observed users searching for the parameter that most highly correlates with output measures. we consequently implemented a simple automated sorting facility that we call "smart sorting". when users click on "smart sort", our method computes the aggregate correlation of each input parameter and all output measures. the parameter with the highest correlation is identified and the data records are sorted by this parameter ). context-sensitive selection during prototyping, users found selection of individual data records too tedious to effectively analyse relationships between input and output. to address this, we developed context-sensitive selection. suppose the cursor intersects row r and column c. in addition to highlighting row r, all directly adjacent rows with the same value for column c also receive focus. for example, compare the highlighted rows in figure where the cursor intersects column one, to figure where it intersects column two. rows in focus are enclosed by a red frame and marked by two red disks in the margins. clicking selects all rows in focus and marks each selected record with blue disks. compound selections are made by multiple selections of this type. clicking on the button labelled "show", below the tabular visualisation at the right, displays all images associated with selected data records in the image browser ). to provide more flexibility, users can rapidly filter records by clicking on "filter" below any column and then specify an interval of interest. all records where the corresponding parameter or measure falls outside the interval are hidden. a strip below each column indicates which parts of its domain are currently displayed. there are situations where users want to look at output images associated with a single data record in real time. by holding the shift key, output images for the table row directly under the cursor are temporarily shown in the image browser. images for a single record can be read and drawn at interactive speeds. design alternatives we also investigated updating the image browser in real time as users select data records, or to use image caching. the former imposes a performance penalty for reading large numbers of images from disk, while the memory footprint of the latter limits scalability. column sorting combined with context-sensitive selection is an effective and efficient way to investigate meaningful subsets of data. we also considered "hard sorting" rows by column then by column and so on. this imposes a column-based hierarchy on the data and forces users to reorder columns to change the hierarchy. instead, our approach lets users choose a column to sort on with one button click. for automated sorting, it is possible to rank all input parameters on their individual correlations and to sort the data by all parameters, in this rank order. however, our users indicated that they prefer sorting by the single most significant parameter and our method therefore implements this approach. providing automation as a "one-click" option, which can be visually verified and easily undone, alleviated fears about added complexity introduced by automated analysis.  in this section, we describe applications of our method by providing an intuitive example and a case study. we also consider further biomedical applications, lessons learned, and opportunities for future work. example: cell nuclei detection cell nuclei detection is an important but challenging step in many high-throughput assays. as demonstrated in additional file s figure shows the results of a cell nuclei detection algorithm for photomicrographs of human ht cells that had been stained to highlight nuclei. the algorithm has five input parameters that had been sampled three times each on meaningful intervals. for each combination of parameter values, the algorithm was run on two input images. for each parameter value combination, object counts for each image were captured as output measures and the outlines of detected nuclei were saved as output images. the results have been sorted on column the nuclei count for the first input image. this shows relationships with the second and fourth parameters. by considering the values of the two parameters in combination with the output measures and images, it is straightforward to identify values for both that produce accurate nuclei detection. these results have been selected in figure a biomedical novice was able to identify these values in less than five minutes. the same task takes more than an hour with conventional parameter tweaking. case study: colour deconvolution for histology histology is the study of tissue at a microscopic scale. tissue is sectioned into micrometre-thin slices, fixed to glass slides, and stained with dyes that highlight different cellular compartments and structures. hematoxylin and eosin dyes are often combined to, respectively, colour cell nuclei blue and cytoplasm and connective tissue pink ). however, different sub-cellular structures and proteins have overlapping absorption spectra, which makes it hard to differentiate contributions of individual dyes. colour deconvolution is an image processing method that can extract individual dyes. this has important biological implications because it allows for quantitative tissue characterisation. for colour deconvolution, a suitable deconvolution matrix must be found. when applied to the original rgb image, this matrix splits it into several images, each representing the contribution of an individual dye. a biomedical image analysis expert had been investigating the assumed optimality of ruifrok and johnson's deconvolution matrix. he had two objectives. develop quality measures to quantify the performance of colour deconvolution. this requires an understanding of the behaviour of the deconvolution method. find optimal values for two corrective parameters to optimise the deconvolution matrix. if non-zero, this would show that ruifrok and johnson's method is not optimal in all cases. the expert had been working on the problem just short of a year when we got involved. the description below of how he achieved o and o with our method was obtained by a diary study and follow-up interview. preparation the expert worked on two tissue types, liver and lymphoma, which had been stained with h&e. by using prior knowledge of the light absorption properties of tissue, the expert had developed two corrective parameters, p and p to apply to the deconvolution matrix and six candidate output measures to quantify the output quality. the first parameter characterizes the haematoxylin stain in absorbing the red, green and blue components of the incident light. this parameter can be used to isolate cell nuclei in most cases. the second parameter characterises the eosin stain and can be used to isolate cytoplasm and connective tissue. respectively, m1-m are the percentage of negative coefficients, mean value of negative coefficients, and standard deviation of negative coefficients corresponding to the first stain, while m4-m record the same results for the second stain. detailed rationale of these parameters and measures are beyond the scope of this paper, but an appreciation of the value of the insights gained with our method does not rely on a specialist understanding. the expert was keen to find an alternative to parameter tweaking and immediately saw the potential of our method. he customised his software to sample the input parameters times each on intervals identified based on domain knowledge. this yielded unique combinations. next, for each of the two input images, colour deconvolution was performed for each unique combination of sampled parameter values, and the corresponding data record was saved to disk. input and output images measured × pixels. visual analysis the expert's data is typical for our application domain. each data record contained two input parameters, two input images, output measures, and four output images. the expert started by analysing the deconvolution results for the liver section image ). o after loading the data, the expert applied smart sorting. this identified and sorted the data on p the parameter with the highest aggregate correlation with the variance of the output measures ). the expert observed a number of patterns suggesting correlations between p and several of the output measures. some appeared to be direct, while others appeared to be inverse relationships. the expert then sorted on p he was surprised by the result shown in figure where clear correlations between the input parameters and the output measures are evident. before using our method, he had only seen single results and had not been able to piece together the relationships shown here. the expert now analysed these relationships ). first, as m and m only took one of two values, they did not appear to carry sufficient information and the expert discarded them, reducing the complexity of the subsequent investigation. next, the expert noticed that m and m appeared to be inversions of each other: when m increases, m decreases and vice versa. finally, output measures m and m also closely resemble each other. the expert concluded that m and m are sufficient to analyse the quality of results. the overview and analysis provided by our method enabled him to understand the behaviour shown by the output measures and, consequently, he was confident in this selection. he confirmed his choice by performing a similar analysis of the results of colour deconvolution on the lymphoma image. figure shows our method with the results of deconvolution applied to both input images. the two input parameters p and p are at the left of the tabular visualisation. at the right is the reduced set of output measures, m and m for the liver images and m1− and m1− for lymphoma. in this way, the expert was able to address o o the results in figure show a direct relationship between the value assumed for parameter p and output measures m and m1− based on the assumed optimality of ruifrok and johnson's deconvolution matrix, the expert expected to find high-quality output when m and m1− approach zero. also, the parameters p and p represent deviations from the original deconvolution row vectors, where their 6th value represents no change. since the default deconvolution matrix was derived from numerous empirical experiments, the expert wanted to veer away from it as little as possible to achieve improved quality output. having established p as the parameter most closely correlated with variation of the output measures, the expert decided to first review those data records closest to where p takes its 6th value and where m and m1- approach zero. figure shows how he selected these records using context-sensitive selection. the expert next reviewed the output images in the image browser. because deconvolution splits each input image into two output images, there are four columns of images in figure the first two columns correspond to the output for the liver section input image while the last two correspond to lymphoma. by reviewing these images, and cross-referencing parameters and measures, the expert identified a combination of input parameter values where p ≠ that yielded higher quality results for both input images than the original deconvolution matrix proposed in. this enabled the expert to address o and show that the ruifrok and johnson deconvolution matrix is not always optimal. reflection we followed up by conducting an unstructured debriefing interview. from this and our analysis of the case study, we conclude the following. first, with our approach the expert was able to effectively and efficiently address his research objectives. in particular, by addressing o our technique led him to a breakthrough in understanding. by discovering that an underlying assumption about the deconvolution algorithm he considered is invalid, he showed that ruifrok and johnson's deconvolution matrix is not optimal for all cases. figure illustrates this for the liver section. second, the total analysis time for both data sets was roughly minutes. in contrast, the expert estimated that an attempt at a similar analysis using his conventional approach would have required several days. third, the expert noted that despite previously focusing on o and o for nearly a year, he had little confidence in the results obtained with his conventional methods. by contrast, he was very confident of the results achieved with our method. in fact, based on his experience, he held a strong conviction that the rigour of analysis that our technique supports is practically unfeasible using his conventional approach. applications by applying the above algorithm to stained histology sections of engineered articular cartilage, scientists at the university of leeds have found a direct correlation between stain intensity, which isolates an extracellular matrix, and the compressive strength of the cartilage. cartilage repair with engineered tissue is an important new regenerative therapy for ageing populations and this approach offers a novel method for quantifying a key property using histology sections that are already routinely taken for subjective inspection. research into regenerative treatment also requires accurate models of, for example, spinal biomechanics. for this, biologists at leeds are using the above algorithm to investigate if the distribution of stains obtained by deconvolution can be used to derive models of the structural properties of intervertebral disks. finally, image processing is a fundamental part of high-content screening workflows. effective and efficient optimisation of these algorithms dramatically reduces the associated time and quality costs. lessons learned the parameter visualisation method described in this paper treats multiple input parameters, input images, output measures, and output images as first-class citizens for the first time. it results from an evolution in our understanding of the problem space. this is mirrored in the progression from our initial work that focuses only on input parameters and output for a single input image, to a limited and makeshift treatment of measures, to the work presented here. the long-term collaboration between us and diverse domain experts has been absolutely essential for this. during this time, our collaborators' understanding and expectations of the role of visualisation also changed. our joint work has convinced them that interactive visualisation is an important analysis paradigm. as our case study shows, visualisation enables them to address their problems in new and more effective ways. by focusing on the problem, and not intrinsically on technical novelty, we were able to achieve a step-change for users. due to the gap in previous parameter visualisation approaches, which are either parameter- or output-centric, they were limited in the types of analysis they could perform. although our approach is partly based on existing methods, it combines these in a novel way to bridge this gap. by documenting the problem space and the design space, we argue that others will also be able to benefit from this work. this echoes calls for design studies by other authors. future work as we show in our case study, our approach increases effectiveness, efficiency and confidence in our application domain, where it is currently very challenging to analyse and understand relationships between multiple inputs and multiple outputs. our approach also has limitations, however. the tabular visualisation was not designed to deal with over-plotting and, in practice, is limited to a maximum of about parameters and a few thousand unique parameter value combinations. this deals with a class of problem that our target users typically face, but will not address all applications of parameter optimisation. for example, population models can contain hundreds of parameters. despite over-plotting, we have successfully analysed just over data records of browsing behaviour from an unrelated usability study, where a total of nine parameters and measures were investigated and where sensitivity plots were viewed in the image browser. this suggests that our approach could scale to larger data sets than designed for and that it has potential for problems outside biomedical image processing. still, our approach requires sample sizes to be chosen in accordance with the number of data records to visualise. we see two ways to cater for scenarios that require greater scalability. first, a brute-force approach is to visualise more samples by using larger displays such as powerwalls, by letting visualisations scroll, or by implementing focus + context techniques. a second approach is computational steering, where visualisation is integrated into a larger iterative cycle aimed at specifying and resampling regions of parameter space on the fly. we can, for instance, envision our visualisation interface integrated into an image processing framework like cellprofiler. more work is required to investigate these possible approaches. another challenge is scaling to very large numbers of input and output images. discussions with experts revealed that they would like to follow up on analyses like our case study with larger-scale validation exercises that involve hundreds or thousands of input images and their corresponding output. this would be valuable to validate the robustness of a set of parameter values. here the emphasis shifts from dealing with the complexity of parameter space to also dealing with the complexity of very large collections of image-based input and output. there are currently no methods that enable users to interactively analyse the output generated for very large numbers of input images and this is an important open challenge for future research.  we presented a visualisation technique for parameter optimisation of biomedical image processing algorithms. it addresses two challenges: dealing with multiple inputs and outputs and enabling understanding of underlying algorithms. to show this, we provided a case study where a biomedical image processing expert used our method for colour deconvolution of histology images. competing interests the authors declare that they have no competing interests. authors' contributions ajp carried out visualisation design and implementation, conducted parts of the case study, and drafted the manuscript. yz provided expert input for visualisation design, carried out and helped to draft the case study. rar helped with visualisation design and to draft the manuscript. all authors read and approved the final manuscript. supplementary material additional file 1 click here for file acknowledgements the authors thank anne carpenter and mark bray, duane carey and scott finlay, and tatiana von landesberger for crucial input. this research was supported by welmec, a centre of excellence in medical engineering funded by the wellcome trust and epsrc. write up was made possible by a leverhulme early career fellowship award to ajp. declarations funding for publication of this article was provided by welmec, a centre of excellence in medical engineering funded by the wellcome trust and epsrc. this article has been published as part of bmc bioinformatics volume supplement 2015: proceedings of the 5th symposium on biological data visualization: part the full contents of the supplement are available online at http://www.biomedcentral.com/bmcbioinformatics/supplements/16/s11
biomedical image processing methods require users to optimise input parameters to ensure high-quality output. first, it is difficult to optimise multiple input parameters for multiple input images. we discuss its application to a colour deconvolution technique for stained histology images and show how it enabled a domain expert to identify suitable parameter values for the deconvolution of two types of images, and metrics to quantify deconvolution performance. the visualisation method presented here provides analysis capability for multiple inputs and outputs in biomedical image processing that is not supported by previous analysis software. we present a visualisation method that transforms users' ability to understand algorithm behaviour by integrating input and output, and by supporting exploration of their relationships.
 biomedical image processing methods require users to optimise input parameters to ensure high-quality output. this presents two challenges. first, it is difficult to optimise multiple input parameters for multiple input images. second, it is difficult to achieve an understanding of underlying algorithms, in particular, relationships between input and output.  we present a visualisation method that transforms users' ability to understand algorithm behaviour by integrating input and output, and by supporting exploration of their relationships. we discuss its application to a colour deconvolution technique for stained histology images and show how it enabled a domain expert to identify suitable parameter values for the deconvolution of two types of images, and metrics to quantify deconvolution performance. it also enabled a breakthrough in understanding by invalidating an underlying assumption about the algorithm.  the visualisation method presented here provides analysis capability for multiple inputs and outputs in biomedical image processing that is not supported by previous analysis software. the analysis supported by our method is not feasible with conventional trial-and-error approaches. visualisationparameter optimisationimage analysisimage processingbiologybiomedicinehistologydesign study10- july 5th symposium on biological data visualization dublin, ireland 