Design and Evaluation ofInteractive Proofreading Tools for Connectomics
Daniel Haehn, Seymour Knowles-Barley, Mike Roberts,
Johanna Beyer, Narayanan Kasthuri, Jeff W. Lichtman, and Hanspeter PfisterFig. 1: Proofreading with Dojo. We present a web-based application for interactive proofreading of automatic segmentationsof connectome data acquired via electron microscopy. Split, merge and adjust functionality enables multiple users to correct thelabeling of neurons in a collaborative fashion. Color-coded structures can be explored in 2D and 3D.
Abstract—Proofreading refers to the manual correction of automatic segmentations of image data. In connectomics, electron microscopy data is acquired at nanometer-scale resolution and results in very large image volumes of brain tissue that require fully
automatic segmentation algorithms to identify cell boundaries. However, these algorithms require hundreds of corrections per cubic
micron of tissue. Even though this task is time consuming, it is fairly easy for humans to perform corrections through splitting, merging,
and adjusting segments during proofreading. In this paper we present the design and implementation of Mojo, a fully-featured singleuser desktop application for proofreading, and Dojo, a multi-user web-based application for collaborative proofreading. We evaluate
the accuracy and speed of Mojo, Dojo, and Raveler, a proofreading tool from Janelia Farm, through a quantitative user study. We
designed a between-subjects experiment and asked non-experts to proofread neurons in a publicly available connectomics dataset.
Our results show a significant improvement of corrections using web-based Dojo, when given the same amount of time. In addition,
all participants using Dojo reported better usability. We discuss our findings and provide an analysis of requirements for designing
visual proofreading software.
Index Terms—Proofreading, Segmentation, Connectomics, Quantitative Evaluation

1

I NTRODUCTION

In computer vision, image segmentation is the process of partitioning
an image into several sub-regions or segments, where individual segments correspond to distinct areas or objects in the image. Automatic
segmentation approaches eliminate user interaction, which is often thebottleneck of manual and semi-automatic segmentation techniques.
However, automatic approaches are computationally expensive andneed to be targeted towards very specific segmentation problems anddata sets to achieve high-quality results. Furthermore, even optimized
automatic segmentation algorithms usually exhibit higher error rates
and are less accurate than manual expert segmentations. Manual pixel

labeling, on the other hand, requires users to have domain-specificknowledge and is usually a tedious and time-consuming process. A
powerful alternative to fully automatic or manual approaches are interactive semi-automatic techniques. They usually rely on minimaluser input to achieve an initial result and allow users to further improve the segmentation by manual adjustments. Proofreading refers
to the manual and semi-automatic correction of automatic segmentations as a post-processing step. In a proofreading tool, segments canbe quickly joined or split to produce the correct segmentation fasterthan it would be possible with manual annotation. The combination of
automatic segmentation and proofreading is the preferred option forlarge data sets where manual segmentation is not feasible.

• Daniel Haehn, Johanna Beyer, and Hanspeter Pfister are with the Schoolof Engineering and Applied Sciences at Harvard University.
E-mail: {haehn,jbeyer,pfister}@seas.harvard.edu.
• Seymour Knowles-Barley, Narayanan Kasthuri, and Jeff W. Lichtman arewith the Center for Brain Science at Harvard University.
E-mail: seymourkb@seas.harvard.edu, bobby.kasthuri@gmail.com,
jeff@mcb.harvard.edu.
• Mike Roberts is with the Computer Graphics Laboratory at Stanford
University. E-mail: mlrobert@stanford.edu.

Our work stems from a collaboration with neuroscientists in thefield of connectomics. Connectomics aims to completely reconstruct
the wiring diagram of the mammalian brain at nanometer resolution,
comprising billions of nerve cells and their interconnections [32, 44].
By deciphering this vast network and analyzing its underlying properties, scientists hope to better understand mental illnesses, learning
disorders and neural pathologies [33]. However, to analyze neuronalconnectivity at the level of individual synapses (i.e., connections between nerve cells), high-resolution electron microscopy (EM) image
stacks have to be acquired and processed (Fig. 2). These image stacks
are typically on the order of hundreds of terabytes in size and often
exhibit severe noise and artifacts. The huge size of these volumes
makes (semi-)automatic segmentation approaches the only viable option, however, the complex structure of the data provides difficulties

Manuscript received 31 Mar. 2014; accepted 1 Aug. 2014; date of
publication xx xxx 2014; date of current version xx xxx 2014.
For information on obtaining reprints of this article, please sende-mail to: tvcg@computer.org.

Fig. 2: Proofreading as part of the Connectome workflow. Electron microscopy data of the mammalian brain gets acquired, registered andsegmented. Since the output of the automatic segmentation algorithm is not perfect, proofreading is a mandatory stage before any analysis.
for automatic segmentation. The resulting segmentations, on average,
require over 120 manual corrections per cubic micron of tissue [30].
Previous research has focused on image acquisition, segmentation [28]and interactive visualization and analysis [16, 8, 7], but little research
has focused on the proofreading stage [38].
A powerful proofreading tool is crucial for enhancing segmentations efficiently and effectively. Particularly, communicating the three
dimensional property of EM stacks is essential for identifying segmentation errors in 3D and for confirming the correctness of changes in theautomatic segmentation. Existing software solutions tend to be geared
towards domain experts and often have a steep learning curve. One
example is Raveler [23], a stand-alone proofreading tool recently developed at the Janelia Farm Research Campus. The huge scale of thedata that needs to be segmented and proofread, however, requires that
proofreading will have to be crowdsourced in the future. This, on theother hand, implies that proofreading will have to be performed in a
distributed setting by non-domain-experts and novice users.
In collaboration with neuroscientists at the Harvard Center for Brain
Science we have developed two novel tools to increase the efficiency,
accuracy and usability for proofreading large, complex EM data. First,
we have developed Mojo–a powerful stand-alone software application [30] for experienced as well as non-expert users that offers advanced semi-automatic proofreading features. Building on the experience we gained from Mojo, we developed Dojo (”Distributed Mojo”)–
a web-based, distributed proofreading application that includes collaborative features and is geared towards non-expert users. Dojo offers
easy access through a web browser, no installation requirements, 3D
volume rendering, and a clean user interface that improves usability,
especially for novice users. To evaluate the effectiveness of Dojo, we
perform a quantitative user study. The study targets non-experts fromall fields with no previous knowledge of proofreading electron microscopy data. We compare Dojo against Mojo and Raveler on a representative sub-volume of connectomics data. The study is designed asa between-subjects experiment with very little training for all participants and a fixed time frame of thirty minutes to proofread the given
dataset. As a baseline, we also asked two domain experts to label thesame sub-volume from scratch using manual segmentation.
Our first contribution is a set of requirements and design guidelines for visual proofreading applications based on our interviews withdomain experts and feedback from an initial deployment of Mojo tonon-expert users, interns and high-school students. Our second contribution is the design and development of Mojo, a stand-alone software application for proofreading. Based on our experiences during
our work on Mojo, we defined requirements for a successive software
which aims at increasing the usability of a proofreading tool geared
towards non-expert users. These thoughts have led to the developmentof web-based Dojo, the third contribution of this paper. Dojo is easier
to use and adds 3D volume rendering and collaborative features. Our
final contribution is our quantitative user study. We present statistically significant results showing that novice users of Dojo are able toproofread a given data set better and faster than with existing proofreading tools. Based on these results we present design guidelines thatwill help developers of future proofreading tools.

2

R ELATED W ORK

Connectomics. Neuroscience and especially connectomics with its
goals and challenges [32] has received a lot of attention recently. Seung [43] motivates the need for dense reconstruction of neuronal structures, underpinning the requirement for segmentation methods that automatically label huge volumes of high-resolution EM images.
Registration, segmentation and annotation for neuroscience. Currently, the main bottleneck for analyzing connectomics data is the needfor registration and segmentation of the underlying image data (Fig. 2).
EM images are acquired slice by slice by physically cutting a block oftissue. Registration has to be performed between slices and also between individual sub-tiles of a single slice. Several methods focus onregistration of large EM data [5, 8, 42].
Segmentation methods for connectomics can be classified into manual [10, 12], semi-automatic [3, 20, 26, 27, 40, 47, 49], and automatic [22, 28, 34, 36, 37, 48] approaches. While initially manual andsemi-automatic approaches were very popular, in recent years the needfor automatic approaches that are scalable to volumes of hundredsof terabytes has become apparent. ITK-SNAP [49] supports semiautomatic segmentation methods based on active contours as well as
manual labeling. Vazquez-Reina et al. [48] propose an automatic 3D
segmentation of EM volumes by taking the whole volume into account rather than a section-to-section approach. They formulate thesegmentation as the solution to a fusion problem with a global context.
Kaynig et al. [28] propose a pipeline for the automatic reconstructionof neuronal processes that is based on a random forest classifier coupled with an anisotropic smoothing prior in a conditional random field
framework and 3D segment fusion. This pipeline is integrated into the
RhoANA open-source software available at http://www.rhoana.org.
Automatic segmentation methods are more scalable than manual orsemi-automatic approaches, but often require a clean-up or proofreading step where incorrect segmentations are fixed. In 2013 the IEEE
ISBI challenge [2] called for machine learning algorithms for the 3D
segmentation of neurites in EM data. The organizers provided test andtraining data, and a manual expert segmentation. While the results
were quite impressive, all algorithms still exhibited a rather high errorrate, motivating the need for proofreading. We use the data of thatchallenge in our user study.
Collaborative segmentation and annotation. EyeWire [3] is an online segmentation tool where novice users participate in a segmentation game to get points for segmenting neuronal structures using asemi-automatic algorithm. D2P [15] uses a micro-labor workforce approach (based on Amazon’s Mechanical Turk) where boolean choicequestions are presented to users and local decisions are combined
to produce a consensus segmentation. Catmaid [41] and the Viking
Viewer [6] are collaborative annotation frameworks for experts, thatallow users to create skeleton segmentations for terabyte-sized data
sets. However, they do not offer proofreading support.
Visualization and visual analytics for connectomics. A good
overview of visualization for connectomics and human connectomics
is given by Pfister et al. [39] and Margulies et al. [35] respectively.
Most visualization frameworks for connectome data have only basic

Fig. 3: User interface of Raveler by Janelia Farm. The interface consists of the 2D slice view (center), the toolbox (right), additional textual information (bottom) and a simple 3D renderer showing boundingboxes of segments (bottom left).
support for 3D rendering [1] and focus on displaying network mapsof connected brain regions [6]. The Connectome Viewer [13] offers ageneral processing and plug-in framework for visualization. Hadwiger
et al. [16] propose a visualization-driven petavoxel volume rendering
framework for high-resolution electron microscopy streams. For visual analysis of connectomics data, interactive query systems havebeen proposed [7, 31]. None of these systems, however, run in a
distributed multi-user environment and many need high-performanceworkstations and modern GPUs for rendering and data processing.
Collaborative and web-based visualization of image data. Ginsburg et al. [14] propose a visualization system for connectome data
based on WebGL [29]. They combine brain surface rendering withtractography fibers and render a 3D network of connected brain regions. The X toolkit [18] offers WebGL rendering for neuroimaging data, and SliceDrop [17] is a web-based viewer for medical imaging data that supports volume rendering and axis-aligned slice views.
Jeong et al. [25] describe an online collaborative pathology viewer.
None of these frameworks, however, support volume rendering of segmented data or interactively updating segmentations.
Proofreading. Until recently, visual proofreading methods for automatically generated segmentations have not received a lot of attention.
Sicat et al. [46] propose a graph abstraction method to simplify proofreading. They construct a skeleton of the segmentation and identify
potential problematic regions in the segmentation and guide users tothese areas. Raveler [23] (Fig. 3), by Janelia Farms, is used for annotation and proofreading and uses a quadtree-based tiling system tosupport large data. It targets expert users and offers many parameters
for tweaking the proofreading process at the cost of a higher complexity. In this paper we introduce two novel proofreading tools: Mojo
(Sec. 4.1) and Dojo (Sec. 4.2); and compare them to Raveler in a quantitative user study (Sec. 5).
3

V ISUAL P ROOFREADING

In this section we introduce the overall workflow for proofreading
high-resolution EM image stacks and discuss common issues (e.g., usability and scalability) before presenting a detailed requirement analysis of the necessary tasks in a scalable proofreading tool.
3.1

Proofreading Workflow

The visual proofreading workflow consists of three main steps:
1. Searching for structures containing segmentation errors.
2. Modifying the existing segmentation to fix the errors.
3. Confirming the correctness of the modified segmentation.

Fig. 4: Segmentation errors. Common errors of automatic segmentation algorithms include merge errors (green), split errors (red) andboundary errors (yellow). The 2D slice visualization is shown on theleft and Dojo’s 3D volume rendering is shown on the right.
The first step – searching for segmentation errors – requires users
to be very focused. This is especially true for volumetric data because tracking 3D structures in a 2D slice-based visualization involves
constant switching between slices. Segmentation data is typically displayed as a colored overlay on top of the original image data. A common strategy to spot segmentation errors is to continuously toggle thevisibility of this segmentation overlay to compare the labeled boundaries to the actual image data. During this search three different typesof errors can be spotted: a) merge errors or under-segmentation, where
two separate structures are erroneously connected; b) split errors orover-segmentation, where a single structure is erroneously split into
several different segments; and c) boundary errors, where the boundaries of a labeled segment do not match the boundaries of the structurein the image data (Fig. 4).
The second step – modifying the existing segmentation – is mentally less demanding than the search phase but needs good tool support, to allow users to quickly and correctly modify the existing segmentation. The most common semi-automatic tools for correcting thesegmentation correspond to the three error types: merge, split, andadjust. Once a segmentation error is spotted, the user chooses the appropriate tool to fix the error. Merge errors can be corrected by using
the split tool to divide the segments. Split errors can be corrected byjoining the segments using the merge tool. Boundary errors can be
fixed by adjusting either the erroneous label or the neighboring labels. In connectomics data, these errors always look similar (e.g., amerge error fails to detect a dark boundary between two lighter structures). Therefore, after understanding these three error types it should
be possible, even for non-expert users, to recognize and subsequently
fix these errors.
The last step – confirming the correctness of the modified segmentation – requires the user to check if the modifications did fix the segmentation error, or if another correction step is necessary. Most proofreading tools only offer a 2D slice view for that task, however, we propose that an additional 3D view can significantly improve the speed
and accuracy in which users can confirm their modifications.
3.2 Proofreading Issues
We identified several issues that have to be considered when designing proofreading tools for connectomics data. The main problems inproofreading tools usually correspond to usability issues, scalability
issues, and failing to target the system towards the expected users, theirneeds and background knowledge. Developers of systems for experts
can expect users to have a thorough domain background as well as ahigher motivation to use the system, hence higher frustration threshold. Most proofreading tools are geared towards expert users and offer
a full-featured segmentation framework and often a very complex userinterface (Fig. 3). When designing a system for non-experts, however,Requirement
R1. Navigation
R2. E. Detection
R3. E. Correction
R4. Validation
R5. Collaboration
R6. Deployment
R7. GUI

Mojo
2D
manualadvanced
2D
n/a
download
complexDojo
2D+3D
manualparameter-free
2D+3D
yes
web accessminimalistic

Raveler
2D+simple 3D
manualvery advanced
2D+simple 3Dn/acompilation
very complex

Table 1: Comparison of proofreading tools. Features offered by
Mojo, Dojo and Raveler are summarized in respect to the requirements
as identified in Section 3.3.

one has to pay particular attention to usability issues and guidance thatwill help novice users to perform their proofreading tasks.
Data set sizes in connectomics are increasing at a rapid pace. Currently our collaborators routinely acquire datasets of several terabytes,
but in the future this will continue to grow. One solution for handinglarge data is to work on sub-volumes of the data. However, it is common that neuronal structures extend across long distances especiallywhen images are in nanometer pixel resolution. These sub-volumes
would then have to be merged in an extra step to guarantee that structures across block boundaries receive corresponding labels. The problem of combining segmentations is further complicated in settings
where multiple users work on the same dataset concurrently. Without a collaboration mode, each user has to either work on a different
sub-volume, or segmentations of different users might overlap. The
former is not scalable to a system with potentially thousands of users
while the latter can result in conflicts where the segmentation between
users differs, and which will have to be solved in a post-processing
step. Collaborative approaches would allow users to discuss unclear
areas or conflicts directly during the proofreading process.
3.3

Requirement Analysis

We regularly met with our collaborating scientists to discuss theirgoals and define required user tasks for proofreading. We conducted
informal as well as semi-structured interviews with them over thecourse of several months and identified several domain-specific proof
reading tasks. After initial development, we installed Mojo at the labof our domain experts, where it was used for a first informal user study.
After this initial testing phase, Mojo was used by a larger group of
non-experts (20+ high-school students and lab interns) to correct thesegmentation of a small data set of a mouse cortex. The feedback acquired in this first deployment step led us to the development of Dojo,targeting a web-based, non-expert, multi-user environment.
We have identified the following general requirements for proofreading large-scale EM image stacks:
R1. Intuitive navigation inside the 3D image stack. Users have tobe able to easily and intuitively navigate within the 3D data including
zooming and navigating within a single slice and multiple slices. A 3Dview is necessary to help non-expert users grasp the spatial relationsin the data and should be able to display either the entire volume or
only selected user-defined structures.
R2. Detection of segmentation errors. To quickly detect segmentation errors the user has to be able to easily switch between different
rendering modes (i.e., showing the raw data, showing the segmentation
data, showing the segmentation outline).
R3. Fast correction of errors. Correcting errors has to be as simple and easy as possible. Manual corrections have to be supported, but
semi-automatic methods for splitting and merging should be the maininteraction metaphors and have to be accurate, fast, and easy to use.
R4. Checking the correctness of modified segmentations. Nonexpert users have to be able to quickly judge the correctness of theirlast modification. This requires visualizing and highlighting the modified segment in 2D as well as in 3D.
R5. Collaborative proofreading environment. The system has tosupport multiple simultaneous users. The amount of data that needs tobe processed necessitates crowdsourcing the proofreading step. This

Fig. 5: User interface of Mojo. The interface consists of the 2D slice
view (center), the toolbox (top) and additional textual information as
well as the label list (bottom, right).
means that multiple users have to be able to work on the same data set
and giving users the ability to work together collaboratively.
R6. Simple deployment. A crowdsourcing system has to support
easy deployment and must not require any special hardware. Highschools, for example, do not allow installing any external software on
their computers. To enable high-school students to work on proofreading, the system has to be web-based and should run in every browser.
R7. Minimalistic GUI. The final requirement targets the usabilityof the system. To support non-expert users, the user interface has to be
simple and easy to understand. It is better to have fewer options thatare well understood by its users, than to have a cluttered user interface
that confuses novice users.
Table 1 summarizes the different features supported by each of theproofreading tools.
4 T ECHNOLOGY
In the following section we describe the technology behind Mojo andDojo. Mojo was developed as a powerful standalone application and is
now used by domain experts and trained researchers to perform proofreading. To overcome the limitations of Mojo regarding installation,
multi-platform support and hardware requirements, we also developed
a web-based proofreading tool. Dojo does not require any installationand can be accessed with a web browser such as Google Chrome andalso runs on tablets and smartphones.
4.1 Mojo - A Standalone Proofreading Tool
Mojo supports the computer-assisted correction of automatic segmentation results by providing a simple scribble interface for correcting
merge and split errors in automatic segmentations. Mojo is capable ofannotating any volumetric segmentation data, but works best when areasonable automatic segmentation can be provided as a starting point.
Mojo is written in C++ and C# and available as open source software
at http://www.rhoana.org.
4.1.1 User Interface
The Mojo GUI (Fig. 5) reflects a typical desktop application, with atoolbar at the top of the window to control interface and view options,a 2D slice view window showing the current segmentation and EM
data, and a segment list showing segment names, ids and sizes.
4.1.2 Proofreading Features
The Mojo interface provides split, merge and adjust tools that enable
edit operations to be performed with simple, wide brush strokes. This
allows annotation to be performed on a standard workstation, without
the use of a tablet drawing or touchscreen device. The interaction
mode of each tool can be customized in the Mojo toolbar.

(a) select(b) click 1(c) click 2(d) click 3

Fig. 7: Merge workflow. Dojo merges segments via mouse clicks.
segmentation tiles but by adding a look-up or redirection entry (e.g.,A→B) into the segment remap table. The remap table is maintained insystem memory on the GPU, so that stored segmentation tile ids canbe loaded directly to the GPU and large volume changes can propagate
quickly to the display. 2D merge, split and adjust operations modify
segment tiles directly and the tile index table is updated accordingly.
4.2
Fig. 6: User interface of Dojo. The interface consists of the 2D slice
view (center), the 3D volume renderer (top right corner), the toolbox(top left corner), additional textual information (bottom left corner)
and an activity log including all connected users (bottom right corner).
The merge tool offers two interaction modes; draw and click. In
draw mode, the user draws a wide brush stroke over the objects to bemerged. Any objects touched by this brush stroke are combined intoa single object. In click mode the user selects a seed object with aleft click and merges additional objects to the seed object with a rightclick. Merging operates on 2D segments, 3D connected objects, or all
3D objects in the volume, depending on the selected merge behavior.
The split tool offers three interaction modes. In boundary line splitting, the user simply draws a wide brush stroke over the membrane that
represents the boundary. From the lines drawn, non-adjacent perimeter pixels are found and used as seed points for a watershed algorithm.
This method is very fast to compute, and results can be adjusted interactively by adding or removing pixels from the boundary line. The
remaining two modes are point-based split, which is similar to a livewire segmentation where individual points are added to the split line;
and region-based split, where seed regions are painted and a watershedboundary is found between them. Once one split has been performed,
Mojo can predict how segments in adjacent slices will be split. User
can navigate through the stack and quickly confirm or modify split
operations while retaining 3D connectivity of the split objects.
In addition to the merge and split tools, the adjust tool allows theuser to manually draw a region and add it to the selected segment.
This tool is useful when a combination of split and merge operations
are required to correct a segment. Segments which have been fully
corrected can be locked, which changes their appearance to a striped
pattern and ensures that they will not be included in further actions.
4.1.3

Visualization

Mojo displays the original data in a single slice view and allows users
to toggle the colored segmentation overlays. Boundaries of segmented
structures can be enhanced by showing contours. Additionally, theuser can zoom in and pan within a single slice or navigate between
slices. When an object is selected, or the mouse hovers over an object,
all parts of that object are highlighted to help the user navigate throughthe volume and to quickly identify all parts belonging to that object.
4.1.4

Dojo - A Distributed Web-Based Approach

Dojo is a web-based proofreading tool consisting of an image server
component written in python and an HTML5/JavaScript based userinterface on the client side. This has the advantage that users can access the software by simply pointing their web-browser to the URL of
the Dojo server. The main goal of Dojo’s GUI and interaction design
is to reduce complexity for non-expert users. Furthermore, Dojo wasdesigned to be compatible with Mojo. Both tools use the same data
structures, ensuring that annotated data from Mojo can be loaded into
Dojo and vice-versa. The source code of Dojo and a demo installation
are available at http://rhoana.org/dojo.
4.2.1

User Interface

The graphical user interface of Dojo (Fig. 6) was designed with nonexpert users in mind and aims to be minimalistic and clean. The 2D
slice viewer uses the full window size while controls, information andhelp are moved to the corners to not disturb the data visualization andto provide a distraction-free environment. All textual information is
kept small but still readable. The elements of the toolbox (i.e., split,
merge, adjust) are presented as simple icons that show help tooltips
upon hovering. Furthermore, to reduce interaction complexity, Dojo’s
proofreading tools are parameter-free and only require simple pointand-click mouse interaction. Additionally, all tools can be activatedwith keyboard shortcuts which are documented in the lower left cornerfor quick reference. The 3D volume rendering view is located in theupper right corner of the main window and can be resized as desired.
4.2.2

Proofreading Features

Dojo provides three proofreading tools, inspired by Mojo’s toolbox
described in Section 4.1.2. The key difference is that Dojo offers asingle interaction mode for each tool, thus simplifying the interface.
With the merge tool, the user clicks on the propagating structure
and then on all segments which should be merged with it (Fig. 7). Thisaction connects the segments across all slices (3D merge) and can also
be used to merge segments which are located on different slices.
The split tool allows users to split a single segment into two or
more segments by drawing a line across the segment that is to be split(Fig. 8). A split is confirmed by clicking in that part of the original

Data Handling

Mojo data is stored on the filesystem as a quadtree hierarchy of 2D image and segmentation tiles. Additional segment information, such as
segment name and size, is stored in an sqlite database. To improve tile
loading and processing times we maintain a tile index table to identify
which tiles each segment id appears in. Additionally, we use a segment remap table to allow fast merge operations. Merging segment A
with segment B can be achieved without having to modify the original(a) select(b) brush(c) confirm(d) result

Fig. 8: Split workflow. Users can split connected segments in Dojo bybrushing over cell boundaries. The software then calculates the best
split which can be accepted or discarded.

(a) select(b) no borders(c) paint(d) result(a)Fig. 9: Adjust workflow. Users can perform fine-grained pixel-wiseadjustments by painting on the image data.
object that should keep the original color. Under the hood, the splitting algorithm works differently to Mojo: Dojo uses all points of thesegment to be split that are not part of the drawn line as seeds for a watershed algorithm, instead of only two perimeter pixels. Additionally,
before computing the watershed algorithm, the original image data isblurred by a Gaussian and then contrast enhanced, which has experimentally been proven to generate better and more stable results.
The adjust tool works as in Mojo and lets users paint on the image
data to extend a segmentation label (Fig. 9).
Merging uses the remap table data structure of Mojo and is computed solely on the client. Once a client adds or removes a merge,
the merge table is sent to the server to keep all clients synchronized.
No pixel data needs to be modified when merging segments. Splitting
is performed on the server and does require pixel modifications. A
split triggers a reload event for all clients to fetch new segmentation
data for the specific slice. Users identify segmentation errors by constantly comparing the original image with the segmentation results.
This tedious procedure results in a high mental workload for users but
Dojo offers hotkeys for switching or adjusting opacity of the layers.
Also, the volume rendering component of Dojo is useful for this task
(Fig. 11). In the future, we want to integrate methods that provide user
guidance by automatically identifying potential errors of such segmentations [37] and showing them to the user.
4.2.3

Visualization and Volume Rendering

In addition to the 2D slice view, Dojo provides full 3D volume rendering of the image stack based on WebGL. We leverage the X toolkit(XTK) for this purpose [18]. Since XTK is primarily used for medical imaging data sets which are smaller in size than EM data, we
extended XTK to support 32 bit label overlays and raw image data oflarger sizes. WebGL does not support 3D textures, therefore, volume
rendering is based on 2D textures [9]. To circumvent memory and texture size limitations, the volume renderer limits the resolution of theloaded image slices to 512 × 512 for the xy-plane. This resolution is
sufficient to display the 3D context of a structure and to gain a better
spatial understanding of the data. The volume rendering can be activated by selecting an icon in the toolbox. Once active, the renderer
displays the full image stack and segmentation volume or multiple selected segmentations. To enhance the users’ orientation, the current
slice position in the image stack is displayed as a red outline. Additionally, we have integrated collaborative features into the 3D viewwhich are explained in more detail in Section 4.2.5 (Fig. 10c). Our
volume renderer is built on top of WebGL and therefore requires a
web browser with WebGL support. Nevertheless, recent advances intechnology have brought WebGL to all major web browsers including
most smartphones and tablets.

(b)

Fig. 10: Collaborative features of Dojo. When active, the collaborative mode of Dojo facilitates proofreading the same data among multiple users. (a) 2D: cursors of other users are visible as colored squares
if working on the same slice, (b) 2D: users can mark difficult regionswith an exclamation mark and (c) 3D: cursors of all connected users
and exclamation marks are visible as colored pins pointing to specificlocations in the volume. Users can directly jump to that location byclicking on the pins.
is well known and similar to Mojo’s quadtree data structure. Unfortunately, most existing web-based image servers exhibit poor performance when loading different zoom levels so the user experience was
not comparable to Mojo or other standalone software applications.
The only framework with a comparable performance was
OpenSeaDragon [4]. OpenSeaDragon is based on the DeepZoom protocol and was initially developed by Microsoft but later open sourced
as a community project. In our initial feasibility study we used an
OpenSeaDragon viewer to connect to an IIP (Internet Imaging Protocol) image server to transfer images in a very performant way. The
initial development stage of Dojo focused on using these technologies and it worked well for raw images. Unfortunately, there was no
easy way of transferring segmentation data nor to display it as overlays. We initially added this functionality to OpenSeaDragon in close
collaboration with their core developers, however, eventually had toabandon this path. While the basic integration of segmentation data
worked, the viewer exhibited severe flickering artifacts and the image
server did not support correct downsampling of segmentation masks.
Another problem was the bit depth: our segmentation data can haveup to 64 bit per pixel, which is not supported by the OpenSeaDragon
framework. Based on these experiences we decided to a) develop ourown client side image viewer and b) develop our own image server.
Using a custom web-client and image server let us optimize thetransfer of images to our specific requirements. We now transfer
high bit-per-pixel segmentation data directly as a zlib-compressed byte
stream. This results in data sizes comparable to PNG encoding without any interpolation errors or loss of precision. Furthermore, the development of a custom image server and web client led to significant
performance improvements. Zooming and scrolling through the image
stack can now be done in real-time, even for large volumes.
4.2.5

4.2.4

Data Handling - Large image data on the web

One key feature of Dojo is the support of large image and segmentation data. Transferring large amounts of image data between server
and web client has previously been explored as part of several research
projects [24, 41]. When designing Dojo we therefore evaluated different frameworks for interactively displaying large-scale images on theweb. Most existing solutions are based on quadtree multi-resolution
hierarchies that always load the currently requested resolution of theimage data–low resolution levels for far-away and zoomed-out views
and high resolution levels when the user zooms in. This approach(c)Collaborative Features

Web pages inherently support connections from multiple clients (i.e.,
web browsers) and thus multiple users at the same time. By default,clients fetch information from and post information to the server. Usually, servers do not push information to all clients. Nevertheless, thelatter can be achieved by using web sockets. We decided to use a web
socket server in addition to standard HTTP to enable synchronization
between all connected clients. Modifications of image data throughmerging, splitting and adjusting are sent to the server which then distributes them to each client. This can be a heavy operation depending
on how many clients are connected. Thus, we limit the transfer toof Dojo with no experience in proofreading and very short training, will perform significantly better (i.e., more accurately) thanusers of Mojo or Raveler.
• H.2 Dojo’s usability is higher than other tools. Users of Dojo
will report increased usability and that they like the system.
• H.3 Given a fixed short time frame, proofreading by nonexperts gives more correct results than completely manual
annotations by experts. In a fixed short time frame, inexperienced participants will generate a corrected data set which ismore accurate and similar to the ground truth than a manual
labeling from domain experts that was done in the same timeframe.
Fig. 11: Volume rendering of proofreading results in Dojo. Prior to
proofreading, incorrect segments are present (different colors). Right:
After proofreading, neurons are smooth and uniformly colored.
coordinates and meta information but never send binary data via web
sockets. Additionally, each image operation gets stored as an entry in
the activity log which is displayed on all connected clients, and locatedat the bottom right corner of the view.
Proofreading of larger volumes can be sped up considerably when
multiple clients are working on the same volume. Hence, in addition
to the synchronization of the current state of segmentation, we decided
to add several collaborative features. Mouse cursors of all connected
users are shared (Fig. 10). If two or more users work on the same tilein the image volume, the other users’ cursors are shown as little colored squares. The colors are randomly assigned and each users’ actionis also colored in the activity log. If the 3D visualization is active, thecursors are also displayed as colored pins or needles pointing to a position of the volume. It is possible to click on a needle to jump to theposition the other user is currently working on. In addition to cursor
sharing, users can actively mark an area of the data to seek help fromother users. When a user sets such a mark, all other users see a large
exclamation mark in the 3D view and a small exclamation mark if they
view the same slice. After navigating to that position and resolving theissue, it is possible for users to mark this problem as solved. Asking
for help and marking an area as fixed is also reported in the activity log.
Since these collaborative features can be distracting, they are optionaland can be toggled by each individual client.
4.2.6 Limitations
Dojo has several limitations due to its minimalistic user interface andweb-based nature (see Table 1): parameter-free operations do not allow complex operations as in Mojo and Raveler, the 3D view is lessversatile than stand-alone GPU approaches due to memory restrictions
resulting in downsampled textures, and the undo stack is limited.
5 U SER S TUDY
We conducted a quantitative user study to evaluate the performanceand usability of three proofreading tools. In particular, we evaluate
how nearly untrained non-experts can correct an automatic segmentation using Mojo, Dojo and Raveler. We designed a between-subjects
experiment and asked the participants to proofread a small dataset in a
fixed time frame of 30 minutes. We used the most representative subvolume of a larger freely available real-world data set where expert
ground truth is available. We recruited participants from all fields withno experience in electron microscopy data or proofreading of such. Asbaseline, we asked two domain experts to label the same sub-volume
from scratch using manual segmentation in the same fixed time frame.
5.1 Hypotheses
We proposed three hypotheses entering the study:• H.1 Proofreading will be measurably better with Dojo compared to other tools. When presented with identical data, users

5.2 Participants
Because we wanted to study how completely inexperienced users perform with the three proofreading tools, we recruited people of all occupations through flyers, mailing lists and personal interaction. Basedon sample size calculation theory, we estimated the study sample
size as ten users per proofreading tool including six potential dropouts [11, 21]. Nevertheless, all thirty participants completed the study(N = 30, 17 female; 20-40 years old, M = 27). Participants had noexperience with electron microscopy data or proofreading of such andhad never seen or used any of the three software tools. They received
monetary compensation for their time.
5.3 Experimental Conditions
The three conditions in our study were the proofreading tools Mojo,
Dojo and Raveler. Each participant proofread the same dataset in atime frame of 30 minutes. The requirement for participating was noexperience in proofreading EM data. Since the three tools run on different platforms (except web-based Dojo which runs on all), we used
three different machines with similar and standard, off-the-shelf hardware. Therefore, the only variable was the used software.
5.3.1 Dataset
We used a publicly available dataset of a mouse cortex(1024x1024x100 pixels) that was published for the ISBI 2013
challenge ”SNEMI3D: 3D Segmentation of neurites in EM images”.
It was acquired with a serial section scanning electron microscope(ssSEM) with a resolution of 6x6x30 nm/pixel. We trained our
automatic segmentation pipeline (Rhoana pipeline) on a similar
dataset and used it to segment the data. Details of the segmentation
pipeline are published in [28]. Manually labeled expert segmentation
was available as a ground truth for the complete dataset. Since it is
not feasible to let users proofread such a large volume, we cropped
a sub-volume to 400x400x10 pixels. To find the most representative
sub-volume (i.e., the sub-volume with the distribution of object sizes
that is closest to the empirical mean distribution of object sizes in thevolume) we calculated object size histograms, used them as featuresin a multi-dimensional vector space and chose the sub-volume withthe minimal Euclidean distance to the centroid.
5.4 Procedure
Each study session started off with the participants signing the consent form and a basic demographic survey (age, sex, occupation, neuroscience background, scientific visualization background, familiarity with proofreading of segmentations and familiarity with EM data).
Next, the participants were introduced to the Connectome project andits typical workflow (Fig. 2). Then, participants sat down at their randomly assigned study station which ran one of the three proofreading
tools and the experimenter explained the tool. To demonstrate merging
and splitting functionalities of each tool, a training dataset was loaded
which was the second most representative sub-volume of the larger
dataset with the same size of the test dataset.
There was no common region between the training data and thetest data even though both were highly representative of the larger
dataset. After explaining two merge and two split operations (averageFig. 12: Variation of information. The VI similarity measure for eachtool after proofreading for 30 minutes (lower is better). Participants
using Dojo achieved a lower VI than subjects with the other tools.
These results are statistically significant. The red line shows the VI of
the automatic segmentation before proofreading. The blue lines show
the VI of the experts’ manual segmentation after 30 minutes.

Fig. 13: Rand index. The RI similarity measure for each tool after
proofreading for 30 minutes (higher is better). Participants using Dojo
achieved a higher RI than subjects with the other tools. These results
are statistically significant. The red line shows the RI of the automatic
segmentation before proofreading. The blue lines show the RI of thetwo experts’ manual segmentation after 30 minutes.

time about 5 minutes), the participants were asked to try the proofreading tool themselves for another 5 minutes. The experimenter thenloaded the test dataset which was the same for each participant. For the
next 30 minutes participants were asked to correct as many segmentation errors as possible but were warned that it was highly unlikely thatthey could finish proofreading the complete dataset in the given timeframe. During the assessment, usage questions regarding the proofreading software were answered in a short manner.
After 30 minutes or if the participant decided to stop the experiment, the participants completed a qualitative questionnaire with ten
questions regarding the software. Then, they answered the raw NASATLX standard questions for task evaluation [19]. At the end of thesession, participants were presented their similarity scores in respect
to the ground truth segmentation and could give general feedback andcomments. The entire study session took approximately 60 minutes.

these dependent variables using analysis of variance followed by parametric tests. For the subjective responses on Likert scales, we createdsub-groups and performed ANOVA according to Holm’s sequentiallyrejective Bonferroni method [45] and parametric tests, if relevant.
6

The results of our user study demonstrate a modest advantage of
Dojo regarding the quality of corrections. The difference in similarity was minor but statistically significant (Sec. 6.1). Another interesting but expected finding was that proofreading yielded better performance than manual segmentation of experts starting from scratch,when given the same short time frame (Sec. 6.3). Overall, participants
reported better usability of Dojo compared to the other proofreading
tools (Sec. 6.4), thus confirming our initial hypotheses.
6.1

5.4.1

Expert Segmentations

To generate a baseline measure regarding hypothesis 3, we asked
two experts to manually annotate the representative sub-volume fromscratch. The experts used their software of choice, ITK-SNAP [49].
We set a time limit of 30 minutes and computed the similarity measures variation of information and Rand index as well as Edit Distance.
5.5

R ESULTS AND D ISCUSSION

Similarity

The initial segmentation for all participants was created using the
Rhoana pipeline and had a VI of 0.98 and a RI of 0.53. The ten participants using Dojo had a mean VI of 0.90 (SD = 0.10). The nine
participants using Mojo had a mean VI of 1.11 (SD = 0.12). The nine
participants using Raveler had a mean VI of 1.12 (SD = 0.25). The ef-

Design and Analysis

We used a single factor between-subject design with the factor proofreading tool (Mojo, Dojo, Raveler). The participants were randomly
assigned to one of the three tools. From our group of participants(N = 30) we excluded two subjects (using Raveler and Mojo). Oneof the two subjects stated to be familiar with proofreading of EM data
and for the other participant Mojo crashed after 20 minutes.
Our dependent measures were two measures of similarity between
participants’ results and the ground truth segmentation (manually labeled by experts, publicly available), the number of not performed
merges and splits to fully correct the segmentation, as well as the participants’ subjective responses recorded on a 7-point Likert scale.
Similarity was measured as variation of information (VI) (Fig. 12)
and Rand index (RI) (Fig. 13) which are common benchmarks for clustering comparison in computer vision. VI is a measure of the distance
between two clusterings, closely related to mutual information, but
lower being better. Rand index is a measure of similarity, related tothe accuracy, meaning higher scores are better. The number of not
performed merges and splits to correct the segmentation is the Edit
Distance, another common metric in computer vision. We calculated
these measures and treated them as continuous variables. We analyzed

Fig. 14: Edit Distance. The ED similarity measure for each tool after
proofreading, in respect of the ground truth segmentation after 30 minutes (lower is better). On average, participants using Dojo reached alower ED than subjects with the other tools. The red line shows the
ED of the automatic segmentation before proofreading. The blue lines
show the ED of the experts’ manual segmentation after 30 minutes.

fect of the proofreading tool, therefore, was significant, F2,25 = 5.04,
p = .015. Post hoc comparisons (after Bonferroni correction) indicatethat the mean VI for results with Dojo was significantly lower than for
Mojo (t25 = 2.06, p = .0411) and also lower than the one for Raveler(t25 = 2.06, p < .01). Figure 12 shows this relation. We also analyzed
the mean RI of participants using Dojo which was 0.55 (SD = 0.03).
For participants using Mojo, the mean RI was 0.51 (SD = 0.02) and forparticipants who used Raveler, the mean RI was 0.5 (SD = 0.06). Thisdifference was statistically significant, F2,25 = 3.59, p = .043. Further
testing (after Bonferroni correction) showed, that only the difference
between Dojo and Raveler was significant (t25 = 2.06, p = .05). The
results are displayed in Figure 13.
6.2 Edit Distance
The edit distance metric (ED) reports how many operations have to be
performed to reach the state of another labeled image. We calculated
the ED for the proofread EM data as the sum of required 2D splits and3D merges to reach the ground truth for each participant. These are theoperations which can be performed by all three tools. The initial segmentation which was the input for all participants, had an ED of 54 (32splits, 22 merges). A blank segmentation of our data had a ED value of
386, requiring 386 splits and 0 merges. The mean ED of participantsfor Dojo was 59.1 (SD = 23.28), for Mojo 62.9 (SD = 12.03) and forRaveler 83.22 (SD = 37.03). These results were not statistically significant but they follow the trend of better performance with Dojo. In
fact, about half of the participants using Dojo were able to reduce theED. Figure 14 shows this relation. Even though the improvement of
the ED was not statistically relevant, the improvements of VI and RI
allow us to accept H1 and to confirm that complete novice users perform slightly better on a proofreading task using Dojo than with othertools. Interestingly, many participants were not able to improve theautomatic segmentation but made it worse. In fact, only participants
using Dojo were able to correct the segmentation on average. We believe that this is caused by several of factors:
1. Three-dimensional thinking. It is hard for untrained users to
grasp the 3D structure of EM data. Dojo provides full 3D volume
rendering to help users get a three dimensional intuition of thedata, for single structures as well as for the whole EM stack.
2. Difficulty identifying boundaries. EM data can be very noisy
and cell boundary detection needs to be practiced. From observations, it seemed that a large amount of time was spent by
participants trying to identify boundaries.
3. Time. Participants tried to correct as much as possible in thegiven time frame. Even though they were told to only perform
corrections when confident, they felt rushed. Therefore, we want
to do a follow-up study without that fixed time frame.
4. Usability of tools. The usability of many existing proofreading
tools is lacking. An overwhelming number of features and parameters are available and confuse users. Only Dojo provides
parameter-free, easy-to-use tools.
5. Almost zero training. The participants of this study received,on purpose, nearly no training regarding the data or the software.
We believe that training in the range of hours or days can drastically improve the performance of non-experts. Especially the
manual detection of errors is very difficult for novices. Algorithms that guide the user to potential errors could greatly improve user performance.
6.3 Proofreading versus Manual Expert Segmentation
The two experts who were asked to manually label the data set fromscratch in 30 minutes, did not reach the result of our proofreading participants starting from automatic segmentation (Figures 12, 13, 14).
The mean values were VI=1.77, RI=0.26 and ED=277. These results were significant in comparison to Dojo (t25 = 2.06, p < .0001).
This is no surprise since manual segmentation is very time consuming.
Hence, we accept H3. In the given time frame experts were able tolabel 60.3% and 57.8% of the sub-volume. It is clear that given moretime, the VI, RI and ED measures would improve rather quickly for
manual expert segmentations.

6.4

Subjective Responses

All subjective responses were recorded on a 7 point Likert scale with
1=fully disagree and 7=fully agree. To ensure representative results,
we grouped the questions and performed Holm’s sequentially-rejective
Bonferroni adjustment (N reported) before reporting statistic significance. We observed statistical significance for qualitative responses
regarding usability: Participants stated that the tool is easy to use
for Dojo M = 4.5 (SD = 1.27), Mojo M = 4.11 (SD = 2.03) and
Raveler M = 3.22 (SD = 1.09) and that the tool is usable for Dojo
M = 5.1 (SD = 1.1), for Mojo M = 4.3 (SD = 1.87) and for Raveler
M = 3.11 (SD = 1.27). After adjustment with N = 2, being usable isstatistically significant (F2,25 = 4.57, p = .0408) while further analysis only confirmed significance between Dojo and Raveler (t25 = 2.06,
p = .006). We asked three questions regarding the visualization components: Participants stated that the slice visualizations were pleasingfor Dojo M = 5.8 (SD = 0.92), Mojo M = 4.7 (SD = 1.73) and Raveler M = 4.33 (SD = 1.58), the segment visualizations were pleasingfor Dojo M = 5.5 (SD = 1.08), Mojo M = 5.11 (SD = 0.93) and Raveler M = 4.0 (SD = 1.5) and additional information beside 2D was
useful for Dojo M = 5.0 (SD = 1.63), Mojo M = 4.0 (SD = 2.2)
and Raveler M = 4.0 (SD = 1.63). Unfortunately, none of these results were significant after adjustment (segment visualization was before). Nevertheless, we do believe that the 3D volume rendering ofDojo had impact on the superior quantitative performance reported in
the previous section. Other interesting observations were that participants reported that the merge tool was easy to use: for Dojo M = 5.8(SD = 1.14), Mojo M = 4.89 (SD = 1.17) and Raveler M = 3.78
(SD = 1.39). After adjustment with N = 3, this was statistically significant (F2,25 = 6.37, p = .0174). The NASA-TLX workload reportedby the users did not yield any interesting results. As the overall conclusion, participants reported that they generally liked the used software for Dojo M = 5.4 (SD = 1.17), Mojo M = 4.33 (SD = 1.66) and
Raveler M = 3.67 (SD = 1.41). This result was statistically significant with N = 1 (F2,25 = 3.62, p = .0416) but further analysis showedsignificance only between Dojo and Raveler (t25 = 2.06, p = .0135).
Because of that and the usability findings, we partially accept H2.
The qualitative as well as the quantitative evaluation was in favor of
Dojo which also matches our observations during the user study.
7

C ONCLUSION AND F UTURE W ORK

In this paper, we have presented an analysis and evaluation of proofreading tools for automatic segmentations of connectomics data.
Based on this analysis and on our experience with Mojo, we developed Dojo, a web-based proofreading tool. Dojo provides several
proofreading aids such as a clean and minimalistic user interface aswell as 3D volume rendering. We performed a between-subjects user
study regarding Dojo, Mojo and another proofreading tool called Raveler. The results of our quantitative evaluation confirm the need foreasy-to-use and well-designed visualization features for proofreading
tools but also show the need of user training regarding the proofreading task. The individual differences between the evaluated tools were
not large due to study design limitations (see Section 6.2).
In the near future we will deploy Dojo to hundreds of high school
students to proofread EM data in a collaborative fashion. Furthermore,
we want to investigate in novel methods for simplifying cell boundary detection. Using interactive edge-detection to highlight boundaries
could significantly improve the performance of non-expert users. We
would also like to perform an in-depth user study without the time limitation to see how far proofreading can optimize faulty segmentations.
Furthermore, we hope that offering an open source proofreading tool
will promote the adoption of web-based scientific visualization andencourage more research in novel proofreading applications.
ACKNOWLEDGMENTS
We would like to thank Steffen Kirchhoff for his input. This work
was partially supported by NSF grant OIA-1125087, the NIMH Silvio
Conte Center (P50MH094271) and NIH grant 5R01NS076467-04.

R EFERENCES[1] Rambo3d. http://openconnectome.github.io/Rambo3D/,
2012. Accessed on 31/03/2014.
[2] IEEE ISBI challenge: SNEMI3D - 3D segmentation of neurites in EM
images. http://brainiac2.mit.edu/SNEMI3D, 2013. Accessed on 31/03/2014.
[3] Eyewire. http://eyewire.org/, 2014. Accessed on 31/03/2014.
[4] OpenSeaDragon. http://openseadragon.github.io/, 2014.
Accessed on 31/03/2014.
[5] A. Akselrod-Ballin, D. Bock, R. Reid, and S. Warfield. Improved registration for large electron microscopy images. In IEEE Int. Symp. on
Biomedical Imaging (ISBI ’09), pages 434–437, 2009.
[6] J. Anderson, S. Mohammed, B. Grimm, B. Jones, P. Koshevoy, T. Tasdizen, R. Whitaker, and R. Marc. The Viking Viewer for connectomics:
Scalable multi-user annotation and summarization of large volume data
sets. Journal of Microscopy, 241(1):13–28, 2011.
[7] J. Beyer, A. Al-Awami, N. Kasthuri, J. W. Lichtman, H. Pfister, and
M. Hadwiger. ConnectomeExplorer: Query-guided visual analysis oflarge volumetric neuroscience data. IEEE Trans. on Vis. and Computer
Graphics (Proc. IEEE SciVis 2013), 19(12):2868–2877, 2013.
[8] J. Beyer, M. Hadwiger, A. Al-Awami, W.-K. Jeong, N. Kasthuri, J. W.
Lichtman, and H. Pfister. Exploring the connectome: Petascale volume
visualization of microscopy data streams. IEEE Computer Graphics andApplications, 33(4):50–61, 2013.
[9] B. Cabral, N. Cam, and J. Foran. Accelerated volume rendering andtomographic reconstruction using texture mapping hardware. In IEEE
Symposium on Volume Visualization, pages 91–98, 1994.
[10] A. Cardona, S. Saalfeld, J. Schindelin, I. Arganda-Carreras, S. Preibisch,
M. Longair, P. Tomancak, V. Hartenstein, and R. J. Douglas. TrakEM2
software for neural circuit reconstruction. PLoS ONE, 7(6):e38011+,
2012.
[11] L. Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments,& Computers, 35(3):379–383, 2003.
[12] J. C. Fiala. Reconstruct: A free editor for serial section microscopy.
Journal of Microscopy, 218(1):52–61, 2005.
[13] S. Gerhard, A. Daducci, A. Lemkaddem, R. Meuli, J. Thiran, and P. Hagmann. The connectome viewer toolkit: An open source framework to
manage, analyze, and visualize connectomes. Frontiers in Neuroinformatics, 5, 2011.
[14] D. Ginsburg, S. Gerhard, J. E. C. Calle, and R. Pienaar. Realtime visualization of the connectome in the browser using WebGL. Frontiers in
Neuroinformatics, (95), 2011.
[15] R. J. Giuly, K.-Y. Kim, and M. H. Ellisman. DP2: Distributed
3D image segmentation using micro-labor workforce. Bioinformatics,
29(10):1359–1360, 2013.
[16] M. Hadwiger, J. Beyer, W.-K. Jeong, and H. Pfister. Interactive volume
exploration of petascale microscopy data streams using a visualizationdriven virtual memory approach. IEEE Trans. on Visualization and Computer Graphics (Proceedings IEEE SciVis’12), 18(12):2285–2294, 2012.
[17] D. Haehn. Slice:drop: Collaborative medical imaging in the browser. In
ACM SIGGRAPH 2013 Comp. Anim. Festival, pages 1–1, 2013.
[18] D. Haehn, N. Rannou, B. Ahtam, E. Grant, and R. Pienaar. Neuroimaging
in the browser using the X Toolkit. Frontiers in Neuroinformatics, 2012.
[19] S. G. Hart and L. E. Staveland. Development of NASA-TLX (Task Load
Index): Results of empirical and theoretical research. In Human Mental
Workload, volume 52 of Advances in Psychology, pages 139 – 183. 1988.
[20] M. Helmstädter, K. L. Briggman, and W. Denk. High-accuracy neurite
reconstruction for high-throughput neuroanatomy. Nature Neuroscience,
14(8):1081–1088, 2011.
[21] W. Hwang and G. Salvendy. Number of people required for usability
evaluation: The 10+-2 rule. Commun. ACM, 53(5):130–133, May 2010.
[22] V. Jain, B. Bollmann, M. Richardson, D. Berger, M. Helmstädter, K. Briggman, W. Denk, J. Bowden, J. Mendenhall, W. Abraham, K. Harris,
N. Kasthuri, K. Hayworth, R. Schalek, J. Tapia, J. Lichtman, and S. Seung. Boundary learning by optimization with topological constraints. In
Proceedings of IEEE CVPR 2010, pages 2488–2495, 2010.
[23] Janelia Farm. Raveler. https://openwiki.janelia.org/
wiki/display/flyem/Raveler, 2014. Accessed on 31/03/2014.
[24] A. Janke, H. Waxenegger, J. Ullmann, and G. Galloway. Tissuestack: A
new way to view your imaging data. Frontiers in Neuroinformatics, (3).
[25] W. Jeong, J. Schneider, A. Hansen, M. Lee, S. G. Turney, B. E. Faulkner-

Jones, J. L. Hecht, R. Najarian, E. Yee, J. W. Lichtman, and H. Pfister. A
collaborative digital pathology system for multi-touch mobile and desktop computing platforms. Comp. Graphics Forum, 32(6):227–242, 2013.
[26] W.-K. Jeong, J. Beyer, M. Hadwiger, R. Blue, C. Law, A. Vázquez-Reina,
C. Reid, J. W. Lichtman, and H. Pfister. SSECRETT and NeuroTrace:
Interactive visualization and analysis tools for large-scale neurosciencedatasets. IEEE Computer Graphics and Applications, 30(3):58–70, 2010.
[27] E. Jurrus, S. Watanabe, R. J. Giuly, A. R. Paiva, M. H. Ellisman, E. M.
Jorgensen, and T. Tasdizen. Semi-automated neuron boundary detection
and nonbranching process segmentation in electron microscopy images.
Neuroinformatics, 11(1):5–29, 2013.
[28] V. Kaynig, A. Vázquez-Reina, S. Knowles-Barley, M. Roberts, T. Jones,
N. Kasthuri, E. Miller, J. W. Lichtman, and H. Pfister. Large-scale automatic reconstruction of neuronal processes from electron microscopy
images. In arXiv: 1303.7186 [q-bio.NC], 2013.
[29] Khronos Group. WebGL specification. http://www.khronos.
org/registry/webgl/specs, 2014. Accessed on 31/03/2014.
[30] S. Knowles-Barley, M. Roberts, N. Kasthuri, D. Lee, H. Pfister, and J. W.
Lichtman. Mojo 2.0: Connectome annotation tool. Frontiers in Neuroinformatics, (60), 2013.
[31] A. Kuß, S. Prohaska, B. Meyer, J. Rybak, and H.-C. Hege. Ontologybased visualization of hierarchical neuroanatomical structures. In Proceedings of Visual Computing for Biomedicine, pages 177–184, 2008.
[32] J. W. Lichtman and W. Denk. The big and the small: Challenges ofimaging the brain’s circuits. Science, 334(6056):618–623, 2011.
[33] J. W. Lichtman, J. Livet, and J. R. Sanes. A technicolour approach to theconnectome. Nature reviews. Neuroscience, 2008.
[34] T. Liu, C. Jones, M. Seyedhosseini, and T. Tasdizen. A modular hierarchical approach to 3D electron microscopy image segmentation. Journalof Neuroscience Methods, 226(0):88 – 102, 2014.
[35] D. S. Margulies, J. Böttger, A. Watanabe, and K. J. Gorgolewski. Visualizing the human connectome. NeuroImage, 80(0):445 – 461, 2013.
[36] J. Nunez-Iglesias, R. Kennedy, T. Parag, J. Shi, and D. B. Chklovskii.
Machine learning of hierarchical clustering to segment 2D and 3D images. PLoS ONE, 8(8):e71715+, 2013.
[37] J. Nunez-Iglesias, R. Kennedy, S. M. Plaza, A. Chakraborty, and W. T.
Katz. Graph-based active learning of agglomeration (GALA): A python
library to segment 2D and 3D neuroimages. Frontiers in Neuroinformatics, 8(34), 2014.
[38] H. Peng, F. Long, T. Zhao, and E. Myers. Proof-editing is the bottleneckof 3D neuron reconstruction: The problem and solutions. Neuroinformatics, 9(2-3):103–105, 2011.
[39] H. Pfister, V. Kaynig, C. P. Botha, S. Bruckner, V. J. Dercksen, H.C. Hege, and J. B. Roerdink. Visualization in connectomics. In
arXiv:1206.1428v2 [cs.GR], 2012.
[40] M. Roberts, W.-K. Jeong, A. Vázquez-Reina, M. Unger, H. Bischof, J. W.
Lichtman, and H. Pfister. Neural process reconstruction from sparse user
scribbles. In Proceedings of MICCAI 2011, pages 621–628, 2011.
[41] S. Saalfeld, A. Cardona, V. Hartenstein, and P. Tomančák. CATMAID:
collaborative annotation toolkit for massive amounts of image data.
Bioinformatics, 25(15):1984–1986, 2009.
[42] S. Saalfeld, A. Cardona, V. Hartenstein, and P. Tomančák. As-rigidas-possible mosaicking and serial section registration of large ssTEM
datasets. Bioinformatics, 26(12):i57–i63, 2010.
[43] S. Seung. Reading the book of memory: Sparse sampling versus dense
mapping of connectomes. Neuron, 62(1):17–29, 2009.
[44] S. Seung. Connectome: How the Brain’s Wiring Makes Us Who We Are.
Houghton Mifflin Harcourt, Feb. 2012.
[45] J. P. Shaffer. Multiple hypothesis testing. Annual Review of Psychology,
46(1):561–584, 1995.
[46] R. Sicat, M. Hadwiger, and N. J. Mitra. Graph abstraction for simplified
proofreading of slice-based volume segmentation. In EUROGRAPHICS
Short Paper, 2013.
[47] C. Sommer, C. Straehle, U. Kothe, and F. A. Hamprecht. Ilastik: Interactive learning and segmentation toolkit. In IEEE Int. Symp. on Biomedical
Imaging: From Nano to Macro, pages 230–233, 2011.
[48] A. Vázquez-Reina, M. Gelbart, D. Huang, J. Lichtman, E. Miller, and
H. Pfister. Segmentation fusion for connectomics. In Proceedings ofIEEE ICCV, pages 177–184, Nov 2011.
[49] P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho,
J. C. Gee, and G. Gerig. User-guided 3D active contour segmentation of
anatomical structures: Significantly improved efficiency and reliability.
Neuroimage, 31(3):1116–1128, 2006.