Reinforcement learning for instance segmentation with high-level priors
Authors: Paul Hilt
Edgar Kaziakhmedov
Sourabh Bhide
Maria Leptin
Constantin Pape
Anna Kreshuk
Venue: NeurIPS 2021 Submitted (2021)
Keywords: segmentation
bioimaging
reinforcement learning
self-supervised

Abstract: Instance segmentation is an important computer vision problem which remains challenging despite impressive recent advances due to deep learning-based methods. Given sufficient training data, fully supervised methods can yield excellent performance, but annotation of ground-truth data remains a major bottleneck, especially for biomedical applications where it has to be performed by domain experts. The amount of labels required can be drastically reduced by using rules derived from prior knowledge to guide the segmentation. However, these rules are in general not differentiable and thus cannot be used with existing methods. Here, we relax this requirement by using stateless actor critic reinforcement learning, which enables non-differentiable rewards. We formulate the instance segmentation problem as graph partitioning and the actor critic predicts the edge weights driven by the rewards, which are based on the conformity of segmented instances to high-level priors on object shape, position or size. The experiments on toy and real datasets demonstrate that we can achieve excellent performance without any direct supervision based only on a rich set of priors.

Stateless actor-critic for instance segmentation with high-level priors
Anonymous Author(s)
Affiliation Address email
Abstract
Instance segmentation an important computer vision problem which remains challenging despite impressive recent advances due to deep learning- Sarning-baseed meth- ods. Given sufficient training data, fully supervised methods can yield excellent performance, but annotation of ground-truth data remains major bottleneck, espe- cially for biomedical applications where it has to be performed by domain experts. The amount of labels required can be drastically reduced by using rules derived from prior knowledge to guide the segmentation. However, these rules are in general not differentiable and thus cannot be used with existing methods. Here, we relax this requirement by using stateless actor critic reinforcement learning, which enables non- non-difien con-dffferentablee rewards. We formulate the instance segmentation problem as graph partitioning and the actor critic predicts the edge weights driven by the rewards, which are based on the conformity of segmented instances high-level priors on object shape, position or size. The experiments toy and real datasets demonstrate that we can achieve excellent performance without any direct supervision based only on a rich set of priors.
5
10
11
12
14
Introduction
18
17 Instance segmentation is the task of segmenting all objects an image and assigning each of them different label. It forms the necessary first step to the analysis of individual objects in scene 19 and is thus of paramount importance in many practical applications computer vision. Over the recent years, fully supervised instance segmentation methods have made tremendous progress both 21 in natural image applications and in scientific imaging, achieving excellent segmentations for very difficult tasks 21
22
23 large corpus training images hard to avoid when the segmentation method needs to take into account the full variability of the natural world. However, in many practical segmentation tasks the appearance the objects can be expected to conform to certain rules which are known a priori. Examples include surveillance, industrial quality control and especially medical and biological imaging applications where full exploitation of such prior knowledge is particularly important as the training data is sparse and difficult acquire: pixelwise annotation of the necessary instance-level groundtruth for a microscopy experiment can take weeks or even months of expert time. The use of shape priors has strong history this domain 3 4 but the most powerful learned shape models
24
25
27
29
31 still require groundtruth [5] and generic shapes are hard to combine with the CNN losses and other,
32 sha-sh priors. For many high-level priors it has already been demonstrated that integration of
33 the prior directly into the CNN loss can lead to superior segmentations while significantly reducing
34 the necessary amounts of training data (6). However, the requirement of formulating the prior as
35 differentiable function poses severe limitation on the kinds of high-level knowledge that can
36 be exploited with such an approach. The aim of our contribution is address this limitation and
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 establish a framework in which a rich set of non-differentiable rules and expectations can be used to
38 steer the network training.
39 To circumvent the requirement of a differentiable loss function, we turn to the reinforcement learning
40 paradigm, where the rewards can be computed from non-differentiable cost function. We base
41 our framework on stateless actor-critic setup 77. providing one of the first practical applications
42 of this important theoretical construct. In more detail, we solve the instance segmentation problem
43 as agglomeration of image superpixels, with the agent predicting the weights of the edges in the
44 superpixel region adjacency graph. Based on the predicted weights, the segmentation is obtained through (non-differentiable) graph partitioning and the segmented objects are then evaluated by the critic, which learns to approximate the rewards based on the object- and image-level reasoning (see Fig.
45
46
47
The main contributions of this work be summarized follows: (i) we formulate instance segmen- tation as a RL. problem based on a stateless actor-critic setup, encapsulating the non-differentiable step of instance extraction into the environment and thus achieving end-to-end learning; (ii) we exploit prior knowledge on instance appearance and morphology by tying the rewards to the conformity of the predicted objects to pre-defined rules and learning to approximate the (non-differentiable) reward function with the critic; (iii) we introduce a strategy for spatial decomposition of rewards based on fixed-sized subgraphs to enable localized supervision from combinations of object- and image-level
49
50
51
52
53
54
55 rules. (iv) we demonstrate the feasibility of our approach on synthetic and real images and show
56 an application to an important segmentation task developmental biology, where our framework delivers an excellent segmentation with no supervision other than high-level rules.
57
2
Related work
58
59
Reinforcement learning has so far not found significant adoption in the segmentation domain. The closest to our work are two methods in which RL has been introduced to learn sequence of segmentation decision steps as Markov Decision Process. In the actor critic framework of [8] the actor recurrently predicts one instance mask at a time based on the gradient provided by the critic. The training needs fully segmented images as supervision and the overall system, including an LSTM sub-network between the encoder and the decoder, is fairly complex. In (9], the individual decision steps correspond to merges of clusters while their sequence defines a hicrarchical agglomeration process superpixel graph. The reward function is based on Rand index and thus not differentiable, but the overall framework requires full (super)pixclwise supervision for training.
60
61
63
64
65
66
67
68 Reward decomposition was introduced for multi agent RL by [10) where global reward is decom-
69 posed into a per agent reward. [ proves that a stateless RL setup with decomposed rewards requires far less training samples than a RL setup with global reward. In [12) reward decomposition is applied both temporally and spatially for zero-shot inference on unseen environments by training on locally selected samples to learn the underlying physics of the environment.
70
71
73 The restriction to differentiable losses is present in all application domains of deep learning. Common
74 ways to address it are usually based on a soft relaxation of the loss that can be differentiated. The
75 relaxation can be designed specifically for the loss, such as, for example, Area-under-Cul Curve [13 for
78 classification or Jaccard Index [4] for semantic segmentation. These approaches are not directly
77 applicable to our use case as we aim enable the use of a variety of object- and image-level priors which can easily be combined without handcrafting an approximate loss for each case. More generally, but still for concrete task loss, Direct Loss Minimization has been proposed for CNN training in For semi-supervised learning of classification or ranking task, Discriminative Adversarial Networks have been proposed as a means to learn an approximation to the loss [6). Most generally, Grabocka et al. in [17 propose to train a surrogate neural network which will serve as a smooth approximation of the true loss. In our setup, the critic can informally be viewed as a surrogate network as it learns to approximate the priors through the rewards by Q-learning.
78
79
80
15
81
83
84
85
Incorporation of rules and priors is particularly important in biomedical imaging applications, where such knowledge can be exploited to augment or even substitute scarce groundtruth annotations.
86
87 For example, the shape prior is explicitly encoded in the popular nuclear [88 and cellular [19]
segmentation algorithms based on spatial embedding learning. Learned non-linear representations
88
89 of the shape are used in [5], while (00 the loss for object boundary prediction made topology-
aware. Domain-specific priors can also be exploited in post-processing by graph partitioning [21]. Interestingly, the energy minimization procedure underlying the graph partitioning can also be
91
incorporated into the learning step [22 23)
2

---

b) Agent
U-Net
3p
GNN
Accumulation
c) Action
L())
a) State
e) Reward
d) Environment
Graph parttooning
R (+)
Superpixess
Figure 1: Interaction of the agent with the environment: (a) shows the state which composed of the image and the superpixel over-segmentation; (b) depicts the agent and the superpixel graph, which accumulates the features for nodes of the GNN from pixels which belong to the corresponding superpixels; (c) given the state, the agent performs the actions by predicting edge weights the superpixel graph; (d) the environment, which includes the graph partitioning built from the weights predicted through agent actions; (e) rewards are obtained by evaluating the segmentation arising from the graph partitioning, based on pre-defined and data dependent rules. The rewards are given back to the agent where they are used for training.
Methods
93
94
The task of instance segmentation can be formalized as transforming an image x into labeling where y maps each pixel to a label value. An instance corresponds to the maximal set of pixels with the same label value. Typically, the instance segmentation problem is solved via supervised learning,
95
96
97
ie. using training set with ground-truth labels ý. Note that y is invariant under the permutation of label values. In general, it is difficult to formulate instance segmentation in fully differentiable manner. Most approaches first predict a "soft" representation with a CNN, e.g. affinities I 24 25),
38
39
boundaries [26] 27] embeddings [28 29 and apply non conn-differentiblle post-pross cost-processing, such as agglomeration [27 30), clustering 32 or partitioning [33,, to obtain the instance segmentation. Alternatively, proposal-baset methods predict bounding-bor box per instance and then predict the instance mask for each bounding-bon 34). Furthermore, the common evaluation metrics for instance segmentation 35 36
101
103
also not differentiable.
105
Our main motivation to explore RL for the instance segmentation task is to cireumvent the restriction to differentiable losses and regardless of the loss to make the whole pipeline differentiable end-to- end even in presence of non-differentiable steps which transform pixelwise CNN predictions into individual instances.
107
109
We formulate the instance segmentation problem using a region adjacency graph where the nodes V correspond to superpixels (homogencous clusters of pixels) and the edges E' connect nodes which belong spatially adjacent superpixels. Given edge weights W, an instance segmentation can be obtained by partitioning the graph, here using approximate multicut solver B7]. Together, the image data, superpixels, graph and the graph partitioning make up the environment of our RL setup. Based on the state of ε, the agent A predicts actions which are used to compute the partitioning. The reward is then computed based on this partitioning. Our agent A is a
EV,
110
112
113
114
115
stateless actor-critic [38] represented by two graph neural networks (GNN) 39 The actor predicts
117 the actions based on the graph and node features F. The node(superpixel) features are computed
by pooling together the corrresponding pixel features based the raw image data.

---

119
Here, we make use of two different setups: Method 1, where the per-pixel features are computed based on the image data with the feature extractor being part of the agent and Method 2 where the feature extractor is part of the environment &. The feature extractor is trained end-to-end in Method whereas it is fixed and thus needs to be pre-trained in Method 2. We use a U-Net [40] as feature extractor and can use hand-crafted features in addition to the learned features. More details about the pre- training can be found in the Appendix. The agent - environment interaction for Method is depicted in Figure I For Method 2 we refer to the Appendix.
120
122
123
124
126
Importantly, this setup enables us use both non- oindefferentiable instance segmentation step and reward function, by encapsulation of the "pixels to instances" step in the environment and learning a policy based on the rewards with a stateless actor critic.
127
128
129
3.1
Stateless Reinforcement Learning Setup
Unlike most RL settings [41] our approach does not require explicitly time dependent state: the actions returned by the agent correspond to the real-valued edge weights in [0, IJ, which are used to compute the graph partitioning. Any state can be reached by single step from the initial state and there exists no time dependency in the state transition. Unlike [9]. we predict all edge values at once which allows us to avoid the iterative strategy of [8 and deliver and evaluate a complete segmentation in every step. We implement a stateless actor critic formulation with episodes of length 1.
131
132
133
135
136
To the best of our knowledge, stateless RL was introduced in 7 to study the connection between generative adversarial networks and actor critics and our method one of the first practical applica- tions of this concept. Here, the agent consists of an actor, which predicts the actions and critic, which predicts the action value Q (expected future discounted reward) given the actions. The stateless approach simplifies the action value function: the action value has to estimate the reward for single step instead of estimating the expected sum of discounted future rewards for many steps. We have explored a multi-step setup as well, but found that it yields inferior results for application; details can be found in the Appendix. As described in detail in 3.2 we compute localized sub-graph rewards instead of relying on single global reward.
137
139
141
143
144
145
The actor corresponds to a single GNN, which predicts the mean and variance of a normal distribution for each edge. The actions a are determined by sampling from this distribution and applying sigmoid to the result to obtain continuous edge weights in the value range [0, 1]. The GNN takes the
146
147
state s = (G, F) as input arguments and its graph convolution for the 2th node is defined as in [39
148
fi = Ya
Σ on (ft. JEN()
149
where 7 as well as th are MLPs, (., .) is the concatenation of vectors and N(i) is the set of neighbors of node i. The gradient of the loss for the actor is given by:
150
V VaLoctor = Vo |SG
Σ #gEG
a
Σ aEag
log(=" (als)) - (s)
(2)
151
This loss gradient is derived following [38] We adapt it to the sub-graph reward structure by calculating the joint action probability of the policy T² over each sub-graph sg in the set of all sub-graphs SG. Using this loss to optimize the policy parameters 0 minimizes the Kullback-Leibler divergence between the Gibbs distribution of action values for each sub-graph Q a) and the
152
153
154
155
policy with respect to the parameters 0 of the policy, a is a trainable temperature parameter which is optimized following the method introduced by 38
156
157
The critic predicts the action value Qeg for each sub- graph sg E SG. It consists of a GNN Qag (s, a) that takes the state = (G, F) as well as the actions a predicted by the actor as input and predicts a feature vector for each edge. The graph convolution from Equation 2 slightly modified:
159
160
f = 1Q
frr (IIINTO))
1
Σ de (fr. (1, JEN(
(3)
161 again 7Q and de are MLPss Based on these edge features Qng is predicted for each sub-graph via an MLP. Here, we use set of subgraph sizes (typically, 6, 12, 32, 128) to generate supervison signal

---

02
10
reward value
0.6
02
1.00
0.90
num obj
0.80
CHT value
80
Figure 3: An example reward landscape Cir- cle Hough Transform (CHT) rewards. High rewards are given the overall number of predicted objects is not too high and if the respective object has a large CHT value. We found exponential gradient of the reward landscape to work best.
The graph is subdivided into sub- graphs, each sub-graph is highlighted by different color. All sub-graphs have the same number of edges (here 3). Overall, we use a variety of sizes covering different notions of locality.
Figure 2:
163
for different neighborhood scales. A given MLP is only valid for a fixed graph size, so we employ different MLP for each size. The loss for the critic is given by:
184
Leritic =
Σ #gEG
1 (0 (s, - r)²
(4)
SSG
Minimizing this loss with respect the action value function's 's parameters 8 minimizes the difference between the expected reward and action values (2 (s,a).
186
167
3.2
Localized Supervision Signals
188
The RL paradigm is to provide global reward for a given state transition 41 However, we find that for our application is possible and desirable instead provide several more localized rewards per state transition: Given large action space with policy represented by complex multivariate probability distribution, is beneficial learn from rewards for the specific actions rather than from scalar global reward for the union of all actions. Of course then requirement arises that the union of local rewards must resemble to the global reward. E.g. the optimal policy is the same for local as for the global reward.
189
170
171
172
175
175
Our actor critic setup (Section 3.1 expects rewards per sub-graph. A good set of sub-graphs should fulfill the following requirements: each sub-graph should be connected so that the information presented the MLP computing the action value for this sub-graph is correlated. The size of the sub-graphs, given by the number of edges, should be a parameter and all sub-graphs should be extracted with exactly that size to serve as valid input for one of the MLPs. The union of all sub- graphs should cover the complete graph so that each edge contributes to at least one action value Qag- The sub- sub-graphs should overlap to provide smooth sum of action values. We introduce Alg Algorithm to extract such a set of sub-graphs (see Appendix). Figure 2 shows the sub-graphs for small example graph.
178
177
178
179
180
181
185
184
Whit While some of the rewards used our experiments can be directly defined for the sub-graphs, most are instead defined per object (see Appendix for details on reward design). We use the following general procedure to map object-level rewards to sub-graphs: first assign to each superpixel the reward its corresponding object, then determine the reward per edge the maximum value of its two incident superpixels' rewards and average the edge rewards to obtain the reward per Here, we use the maximum because high object scores indicate that all actions contributing the respective object should get high reward. However, for low object scores it is not possible to localize specific action responsible for the low score. Hence, by taking the maximum we assign higher score to edges whose incident superpixels belong to different objects, because they probably correspond to correct split. Note that the uncertainty the assignment of low rewards can lead to noisy reward signal, but the averaging of the edge rewards over the sub-graphs and the overlaps
185
186
187
189
191
193
5

---

195
between the sub-grap graaps smooth and partially denoise the rewards. We have also explored a different actor critic setup that can use object level rewards directly, eliminating the need for the sub- sub-graph extraction and mapping. However, this approach yields inferior results, see the Appendix for details.
196
197
4
Experiments
198
199
The agent of our setup acts on the superpixel graph and thus depends on the features assigned the nodes of the graph. We introduced two variants of our algorithm: in the base variant (Method 1) we start from random features and make them part of the agent, allowing them to change through back- pook-propagationn (Fig. [. In contrast, Method 2 acts on predefined features which are provided
200
201
202
as part of the environment and are computed before training, e.g. through unsupervised clustering.
203
204 A very accurate clustering in the features produces an easy problem for the agent to solve where even a global reward for all actions might be sufficient. However, in a real-world setting with no supervision, the noisier the features become the more local the reward has to be. We evaluate Method 2 on synthetic data where self-supervised pretraining can deliver noisy, but meaningful node features.
205
207
Our full setup with Method 1 is evaluated on dataset from light microscopy experiment, where highly regular object shapes are to be expected, but no good feature pre-training is possible.
209
210
To transform the edge weight predictions of the agent into an instance segmentation we use the Multicut [4 algorithm. Here, other options are also possible such as hierarchical clustering used in
211
212
19. but we choose the Multicut for its global optimality property. Hyperparameters of the pipeline
213
were found by grossvallidation (see Appendix).
214
4.1
Synthetic dataset: circles structured ground
215
To evaluate the feasibility of our approach, we create a synthetic dataset with prominent structured background. Our aim is to segment irregular disks on such background using only rule-based supervision. We generate the superpixels by the mutex watershed algorithm [25] which we run on the Gaussian gradient image. The node features of the superpixel graph were computed through sulf-supe pretraining with contrastive loss as described in Appendix and fixed as part of the environment.
217
219
221
As we aim to segment disks, we compute the circularity of the segmented objects for the rewards using the Circle Hough Transform (3). This object-level reward is combined with the global rough estimate of the number of objects in the image to create the reward surface depicted in Fig. 3 The reward for the number of objects provides useful gradient during early training stages: for example, when too few potential objects are found in the prediction, a low reward can be given to what is thought to be the background object. On the other hand, if too many potential objects are found, low reward can be given to all the foreground objects with a low CHT value.
222
223
225
227
228
In more detail, the object rewards fy are composed as follows. We define a threshold on the CHT value (1 = 0.8 in the reward surface shown in Fig. B. Let E [0, 1] be the CHT value corresponding
229
230
to the object and let k be the total number of objects that we expect and be the number of predicted
231
objects. Then
o (f -0556) 0.4, i
Flocal
(5)
0,
otw
Fezp 0.6,
kk)
if 2 k
Tgloba!
(6)
otw
Vfg
Plocat + Yglobal
Here (-)) is the sigmoid function. The input to the sigmoid function normalized to the interval
232
233
3, 3] which was empirically found to be good range. The rewards are always in [0, 1] here this is
split up into jo, 0.5) for the local reward as well as for the global reward.
234
For the largest predicted object we strongly suspect the background object. For this object background rewards Thg are calculated by
236
if S k (repp (1).
rbg
otw
237
Note that this rewards have large globally calculated part which makes this setup not fit for Method 1. It needs some feature representation that already gives a good idea for the clustering, The only

---

Ground
Superpixels
Actions
Ground
PCA Features
Superpixel Edges
Prediction
Rewards
Prediction
MWS baseline
(a) Reinforcement learning output.
(b) Mutex watershed baseline.
Figure The "Circles" dataset. Top left right: ground truth segmentation, raw data, superpixel over- segmentation and a visualizataion for the actions on every edge, where merge action is displayed in green and a split action in red. Bottom left right: the pre-trained pixel embeddings projected to their first 3 PCA components shown RGB, an edge image of the superpixels, the segmentation resulting from the graph agglomeration on the predicted edge weights and a visualization of the rewards based on the CHT, where light green shows high rewards and dark red low rewards.
239
useful local information in the reward is the CHT value. Therefore, the features have fairly distinct structure for circles, the agent should be able to find and to correctly cluster them.
241
Fig. 4 shows the output of all algorithm components on a sample image. For comparison, we also computed mutex watershed [25] predictions. Texture within objects and structured background are inherently difficult for eegion-growing algorithms, but our approach can exploit higher-level reasoning along with low-level information and achieve good segmentation.
243
4.2
Real dataset: light microscopy imaging
Biomedical applications often require segmentation of objects of known morphology which positioned in regular patterns, while extensive prior knowledge is available on variability of both under normal experimental conditions 44. Such data presents the best the reward function can leverage the known characteristics of individual object shape and texture and the overall similarity of the objects.
247
for algorithm as
249
250
251
The dataset used for this experiment contains 317 images extracted from a video of a developing fruitfly embryo acquired with light-sheet microscope [45] (Fig. 5). The image shows boundaries (plasma membranes) of the embryo cells. Across the dataset, 10 images were fully segmented by an expert, we use those for validation.
253
255
Fruitfly embryo is well-studied system for which we can exploit the prior knowledge on the expected cell shape and the radial pattern of cells. Furthermore, as the analysis of cell shape dynamics is a paramount part of many biological experiments, multiple pre-trained networks are available for the cell segmentation task [18 46 477. Due to the differences in sample preparation and image acquisition settings, none of these would work out-of-the-box for our data. However, the CNNs in 47 which are trained to predict boundaries in confocal microscope images of plant tissue, can serve as a strong edge detector to create superpixels in our images. The superpixels are obtained using the seeded watershed algorithm seeds at the local minima of the predicted edge map.
256
257
259
281
263
The rewards for this experiment are designed follows: we set high reward for merging the superpixels which are certain to lie in the background (close to the image boundary or the image center). For the background edges near the foreground area we modulate the reward by the circularity
264
265
of the overall foreground contour. Finally, for the edges which are likely to be in the foreground
266
257
we compute object-level rewards by fitting a rotated bounding box to each object and comparing its
side lengths as well as its orientation to predefined template values. We do not perform semantic segmentation to define precise foreground/backkroumm boundaries, but instead use soft weighting
269
scheme with Gaussian weights to combine object and background rewards based the prior
7

---

Ground Truth
Raw
Edge Prediction
Superpixels
Actions
V
PCA Features a)
b)
Prediction
Rewards
Figure 5: Microscopy dataset experiment. Top left to right: ground truth segmentation; raw data; edge map; superpixel over-segmentation: visualization for the actions on every edge, where a merge action is displayed in green and split action in red. Bottom left to right: a) handcrafted features; b) learned features accumulated on superpixels; c) learned features projected to their first PCA components shown as RGB; the segmentation resulting from the Multicut the predicted edge weights; visualization of the rewards, where light green shows high rewards and dark red low rewards.
271
knowledge the embryo width. An image the weights for different locations in the image can be found in the appendix.
More formally the edge rewards Tedge are calculated as follows. For each edge, we define the distance 273
h between the edge and the center of the image the average distance of the incident objects' center of mass and the center of the image. j the approximate radius of the circle that lies within the
275
foreground and is the maximal distance between and the image boarder. Let further XC(-) be the
277
Gaussian kernel function. Then Tedge yields
K
[h-ol|
(1 a),
if
Tbg
(9)
K.
In
kk
(1 a),
otw
11
jll
rfg
K0
max(rat, To2)
(10)
8
Tedge rg + Tbg
(11)
278 Here Y, 17, 8 are normalization constants. Equation 9 first determines the background probability for
279 an edge by the kernel values. a constitutes reward that directly favors merges which scaled by the background probability. For each edge, Fol and r.2 are the rewards corresponding to the two objects connected to that edge. The object rewards are given by fitting rotated bounding box the object and then compare rotation and dimensions to template values.
281
283
Note that in this experiment no self-supervised pretraining is used for the node features in the agent's GNNs. Unlike the "Circles" dataset, all objects in these images have very similar intensity distributions and can only be separated through the detection of boundaries between them. Instead of the pretraining, we experiment with using few hand-crafted features like the polar coordinate of the node's 's respective superpixel's s center of mass with respect to the coordinate system sitting at the center of the image as well as the superpixel's mass, and with learning other features by back propppropagation from the agent. The handcrafted features are normalized, concatenated to learned features and used as input to the GNN. The projection of the first 3 PCA components of these features into RGB space shown in Fig- respectively for learned feature maps, their projection to node features through the accumulation procedure and finally the concatenation of those and the
285
287
289
291

---

Table Quantitative evaluation on the microscopy dataset. Note that the projection of superpixels to the ground truth (sp gt) sets an upper (lower for VI) bound for our method. We use Symmetric Best Dice as well as the Variation of Information metric to compare all results on the validation set.
Method
SBD
VI merge
VI split
sp gt
0.656 0.019
0.672 = 0.061
0.594 0.028
ours augmentation noise
0.508 0.031
1.233 + 0.156
1.060 0.258
ours
0.482 + 0.020
0.839 + 0.118
1.374 + 0.357
ours without edges
0.446 0.041
0.953 = 0.212
0.994 0.200
ours only handcrafted
0.408 = 0.087
0.987 = 0.101
1.536 + 0.410
edge + mc [47
0.283 = 0.023
3.019 = 0.040
0.342 0.045
contrastive 22
0.215 = 0.009
1.155 = 0.037
3.285 + 0.084
contrastive + edge 28
0.248 = 0.014
1.229 = 0.048
3.336 + 0.073
293
handcrafted features. Note that the learned features converge to a representation which resembles a semantic segmentation of boundaries in the image.
284
295
We train the complete setup for Method end-to-end on a Nvidia GeForce RTX 3090 GPU for days. For comparison we keep the model which achieved the highest reward on the test set. This makes training as well as the validation independent from ground truth annotations. The evolution of the rewards on the validation set for different random seeds is shown in the Appendix. All of the conducted trainings show a stride for high rewards regardless of different random seeding.
297
299
300
For the validation scores we use the variation of information (VI) for both input combinations (merge and split) and the Symmetric Best Dice score. To show the influence of the imperfect superpixels on the final clustering, we project the superpixels to their respective ground truth clustering ("sp gt" in Table 1) which sets an upper (lower case of VT) bound for our method. In this study we use several versions of our approach. In Table [ (ours) refers to method 1 as described in section
301
303
304
(ours + augmentation noise) is the same method but add some noise to the input data during training, (ours without edges) is our method but without the additional edge prediction as input and (ours only handcrafted) is our method where we only use the handcrafted features as described in section 4.2
305
307
We find that learned features significantly contribute to the performance of our method.
309
We compare to the following baseline approaches: edge + mc, which solves the Multicut graph partitioning based on edge weights derived from boundary predictions used for superpixel creation, contrastive, which predicts pixel-wise embedding space that is clustered into instances using
311
312
k-means and for which the embeddings are trained using the discriminative loss function of (88 on the ovules dataset from [47] and contrastive + edge, which is similar to contrastive, but receives the boundary predictions as additional input channel.
313
47
5
Discussion
316
We introduced an end-to-end instance segmentation algorithm which can exploit con-differentiable loss functions and high-level prior information. Our RL approach is based on stateless actor-critic and predicts the full segmentation at every step, allowing us to assign rewards to all objects and reach stable convergence. The segmentation problem is formulated as graph partitioning: we design a reward decomposition algorithm which maps object- and image-level rewards to sub-graphs for localized supervision.
317
318
320
322
We performed proof-of-concept experiments to demonstrate the feasibility of our approach on synthetic and real data and showed in particular that our setup can segment microscopy images with no direct supervision other than high-level reasoning. In the future, we plan to explore other problems and reward functions as well as semi-supervised setup (briefly introduced in Appendix)
323
324
325
326
where we think our approach can be very beneficial. Furthermore, even in case of full supervision with ample groundtruth, our RL-based formulation enables end-to-end instance segmentation with direct levect-level reasoning, which will allow for post-processing-an training of the CNN which predicts object boundaries embeddings.
328

---

References
330
331
[1] Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, and H Sebastian Seung. Superhuman accuracy on the snemi3d connectomics challenge, arXiv preprint arXiv: 100.00700000000, 2017.
332
333
[2] Liang-Chich Chen, Huiyu Wang, and Siyuan Qiao. Scaling wide residual networks for panoptic segmentation, 2021.
334
335
[3] S. Osher and N. Paragios. Geometric Level Set Methods in Imaging, Vision, and Graphics. Springer New York, 2007. ISBN 978038711106.. URL httpt:t/bbookk.... http://///ooooooogolleoo wooggle.d//boooe
338
337
Td=ZNzrB=AAQBAJ
338
[4] Ricard Delgado-Gonzalo, Virginie Uhlmann, Daniel Schmitter, and Michael Unser. Snakes on a plane: A perfect snap for bioimage analysis. IEEE Signal Processing Magazine, 32(1):41-488
339
340
2014.
[5] Ozan Oktay, Enzo Ferrante, Konstantinos Kamnitsas, Mattias Heinrich, Wenjia Bai, Jose Caballero, Stuart A. Cook, Antonio de Marvao, Timothy Dawes, Declan O Regan, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Anatomically constrained neural networks (acnns): Application to cardiac image enhancement and segmentation. IEEE Transactions on Medical Imaging, 37():384 395, 2018. doi: 10. 1 1097TMM.00117.7744444.
342
343
344
346
[6] Hoel Kervadec, Jose Dolz, Meng Tang, Eric Granger, Yuri Boykov, and Ismail Ben Ayed. Constrained-cnn losses for weakly supervised segmentation. Medical Image Analysis, 54:88-99, 2019. ISSN 1361-8415. doi: https://doi.rr//0..00100
348
77] David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. CoRR, abs/1610.01945, 2016. URLpttpp:/arxiv..o http://axxxv..orr////////// 01945
350
[8] Nikita Araslanov, Constantin Rothkopf, and Stefan Roth. Actor-critic instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
352
354
[9] Viren Jain, Srinivas Turaga, Kevin Briggman, Moritz Helmstaedter, Winfried Denk, and Hyun- june Seung. Learning agglomerate superpixel hierarchies. Advances in Neural Information Processing Systems, 24,0 01 2011.
356
[10] Peter Sunehag. Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z.. Leibo, Karl Tuyls, and Thore Graepel. Valoo-deco networks for cooperative multi-agent learning, 2017.
358
360
[1I] Drew Bagnell and Andrew Ng. On local rewards and scaling distributed reinforcement learning. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems, volume 18. MIT Press, 2006. URL httphttt///pppcceediige inw.urreeee........irr.p.pp.pppp neurips. wwwrips.ccppppeprrrrr
381
382
383
384
[12] Huazhe Xu, Boyuan Chen, Yang Gao, and Trevor Darrell. scoring-aggregating--i Learning task-agnostic priors from interactions and sparse rewards for zero-shot generalization. CoRR, abs/1910.08 143, 2019. http:://aaxxi org/ abs 1910. 08143
385
386
367
[13] Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan Rifkin, and Gal Elidan. Scalable learning of non- condddcomposable objectives. In Artificial intelligence and statistics, pages 832-840. PMLR, 2017.
368
389
370
[14] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: tractable surrogate for the optimization of the interecttion-oeerumme measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 1413-4421 2018.
371
372
373
[15] Yang Song, Alexander G. Schwing, Richard S. Zemel, and Raquel Urtasun. Training deep neural networks via direct loss minimization. International Conference on Machine Learning, 2016.
375
10

---

377
[16] Cicero Nogueira dos Santos, Kahini Wadhawan, and Bowen Zhou. Learning loss functions for semi-supervised learning via discriminative adversarial networks, 2017.
378
[17] Josif Grabocka. Randolf Scholz, and Lars Schmidt- Thiemee. Learning surrogate losses, arXiv preprint 1rXiv: 10108, 2019.
380
[88] Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers. Cell detection with star-convex polygons. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 265-273. Springer, 2018.
382
383
[19] Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu. Cellpose: generalist algorithm for cellular segmentation. Nature Methods, 18(1):100-106, 2021.
385
[0)] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. Topology-preserving deep image segmentation. In Advances in Neural Information Processing Systems, volume 32, 2019. URL httpps://prooeeednngs www.nneurppp.
387
388
htw.:ocedddnng.nnuirs......o htt../rrpp...c/app////////////// pdf
389
390
[21] Constantin Pape. Alex Matskevych, Adrian Wolny, Julian Hennies, Giulia Mizzon, Marion Louveaux, Jacob Musser, Alexis Maizel, Detlev Arendt, and Anna Kreshuk. Leveraging domain knowledge to improve microscopy image segmentation with lifted multicuts. Frontiers in Computer Science, 1:6, 2019.
392
394 [22] Jeremy B Maitin- Shepard, Viren Jain, Michal Januszewski, Peter Li, and Pieter Abbeel. Combi- natorial energy learning for image segmentation. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Cur- Associates, Inc., 2016. URL httpsttts:: httptppp://prroceiingg.... http:/urr..p..ccppppp/////////// 31877746660022777558222268888868
396
398
pdf
389
[23] Jie Song. Bjoemn Andres, Michael J Black, Otmar Hilliges, and Siyu Tang. End-to-end learning for graph decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10093-10102, 2019.
400
401
[24] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 642-651, 2019.
403
404
405
[25] Steffen Wolf, Alberto Bailoni, Constantin Pape, Nasim Rahaman, Anna Kreshuk, Ullrich Köthe, and Fred Hamprecht. The mutex watershed and its objective: Efficient, parameter- free graph partitioning. IEEE transactions pattern analysis and machine intelligence, 2020.
408
408
[26] Thorsten Beier, Constantin Pape, Nasim Rahaman, Timo Prange, Stuart Berg. Davi D Bock, Albert Cardona, Graham W Knott, Stephen M Plaza, Louis K Scheffer, et al. Multicut brings automated neurite segmentation closer to human performance. Nature methods, 14(2):101-102, 2017.
410
412
[27]
Jan Funke, Fabian Tschopp, William Grisaitis, Arlo Sheridan, Chandan Singh, Stephan Saalfeld, and Srinivas C Turaga. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE transactions on pattern analysis and machine intelligence, 41(7):1669-1166 1680, 2018
413
414
415
[28]
Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with discriminative loss function. arXiv preprint arXiv:1708.0255/1 2017.
417
418
[29]
Davy Neven, Bert De Brabandere, Marc Proesmans, and Lue Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8837-8845, 2019.
419
420
[30] Alberto Bailoni, Constantin Pape, Steffen Wolf, Thorsten Beier, Anna Kreshuk, and Fred A Hamprecht. A generalized framework for agglomerative clustering of signed graphs applied to 2019.
422
instance segmentation. arXiv preprint

---

424
[31]
Leland McInnes and John Healy. Accelerated hierarchical density based clustering. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 33 33 42. IEEE, 2017.
425
427
[32]
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence, 24(5):603-619, 2002.
428
429
[33] Bjoemn Andres, Thorben Kroeger, Kevin Briggman, Winfried Denk, Natalya Korogod, Graham Knott, Ullrich Koethe, and Fred A Hamprecht. Globally optimal closed-surface segmentation for connectomics. In European Conference on Computer Vision, pages 778-791. Springer, 2012.
430
431
[34] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
434
435
[35] Marina Meilä. Comparing clusterings by the variation of information. In Learning theory and kernel machines, pages 173-187. Springer, 2003.
[36]
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846880) 1971.
438
439
[37]
Brian W Kernighan and Shen Lin. An efficient heuristic procedure for partitioning graphs. The Bell system technical joumall 49(2):291-307 1970.
[38]
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905, 2018. UtLhtt:://arrr
442
org/ /bb/
444
1812. 05905
445
[39]
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, abs//774..01212, 2017. URL http
447
//arxiv. org/abs.////...........oo....s 1704. 01212
448
[40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015. http://///axxx org/
450
abs/1505. (55/15555005977
451
[41] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http:p//hccompleeeieeeeesee/srbs html
452
2nd
453
454
[42] Jörg Hendrik Kappes, Markus Speth, Björn Andres, Gerhard Reinelt, and Christoph Schn. Globally optimal image partitioning by multicuts. In Yuri Boykov, Fredrik Kahl, Victor Lempitsky, and Frank R. Schmidt, editors, Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 31-44, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 118-3-642-2309443
455
456
457
458
459
[43] Allam Shehata Hassanein, Sherien Mohammad, Mohamed Sameer, and Mohammad Ehab Ragab. survey on hough transform, theory, techniques and applications. CoRR, fobs/150..0/// 2015. http::t////axi org/ abs 1502. 02160
460
482
[44] D' Arcy Wentworth Thompson. On Growth and Form. Canto. Cambridge University Press, 1992. doi: 10.10177CB8977810073353
463
464
[45]
Sourabh Bhide, Ralf Mikut, Maria Leptin, and Johannes Stegmaier. Semi Semi-automatic generation of tight binary masks and non-convex isosurfaces for quantitative analysis of 3d biological samples, 2020.
485
486
467
[46] Lucas von Chamier, Romain F Laine, Johanna Jukkala, Christoph Spahn, Daniel Krentzel. Elias Nehme, Martina Lerche, Sara Hernández-Pérez, Pieta K Mattila, Eleni Karinou, Séamus Holden, Ahmer Can Solak, Alexander Krull, Tim-Oliver Buchholz, Martin L Jones, Loie A Royer, Christophe Leterrier, Yoav Shechtman, Florian Jug, Mike Heilemann, Guillaume Jacquemet, and Ricardo Henriques. Democratising deep learning for microscopy with ZeroCostDI AMic.. Nature Communications, 4 2021.
468
469
471
12

---

473
[47] Adrian Wolny, Lorenzo Cerrone, Athul Vijayan, Rachele Tofanelli, Amaya Vilches Barro, Marion Louveaux, Christian Wenzl, Sören Strauss, David Wilson-Sánchez, Rena Lymbouridou. et al. Accurate and versatile 3d segmentation of plant tissues at cellular resolution. Elife, 9: e57613, 2020.
474
475
476
477
[48]
Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function, 2017.
479
[49] Reuben R. Shamir, Yuval Duchin, Jinyoung Kim, Guillermo Sapiro, and Noam Harel. Continuous dice coefficient: a method for evaluating probabilistic segmentations. CoRR, 0ts/1906.11031, 2019. URL http://arxiv. org/ htt////s///19///. 11031
480
Checklist
482
483
1. For all authors...
494
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
486
(b) Did you describe the limitations of your work? [Yes]
487
(c) Did you discuss any potential negative societal impacts of your work? [No] It does not have any negative societal impacts.
488
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
490
491
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not claim any theoretical assumptions.
493
494
(b) Did you include complete proofs of all theoretical results? [N/A] See above.
495
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Yes, see the Supplementary.
497
499
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See the Supplementary
500
501
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] The bars are reported w.r.t. to the samples within the best seed, see Supplementary section for further details
502
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
505
506
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
507
(a) If your work uses existing assets, did you cite the creators? [Yes]
508
(b) Did you mention the license of the assets? [Yes]
509
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
The link to the biological dataset is given in the Supplementary
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? The dataset was provided from the other laboratory as part of collaboration.
512
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No human data is used
515
516
5. If you used crowdsourcing or conducted research with human subjects...
517
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
518
519
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
520
521
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "z2IjFXfW1q", "reviewer": "Reviewer_xg8s", "summary": "This paper presents a RL approach for instance segmentation which formulates the instance segmentation problem as graph partitioning and the actor critic predicts the edge weights driven by the rewards, which are based on the conformity of segmented instances to high-level priors on object shape, position and size. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a novel idea of applying RL methodology to instance segmentation. The stateless actor-critic reinforcement learning theme enables non-differential rewards which could incorporate prior knowledge to guide the segmentation, and relaxes the heavy requirement of pixel-wise labeling. The idea is unique and quite interesting.\nThe related work looks sufficient to me. The experiments are well designed with convincing results.\nThe writing is overall clear, but there are a few parts which could be improved:\n1. For Section3, it's not clear to me the essential difference between Method 1 & 2. It'd be nice to include a graph as Figure 1 to demonstrate Method 2, instead of only including it in Appendix.\n2. There is a lot of valuable information in Appendix. It'd be nice to figure out a way to shorten some parts of Method section (such as Section 3.2 and experiments) and include some content from Appendix here.\n3. For the experiments, some failure cases and analysis are necessary. It'd be good to add a section of ablation studies. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "of3OMesGiV", "reviewer": "Reviewer_iAQ6", "summary": "The paper proposes the use of RL for instance segmentation. In particular, the paper uses super pixels together with graph partitioning and RL to learn the labels of instances. The model is validated on two datasets, one synthetic and one biomedical imaging dataset and is numerically compared to two approaches ([28] and [47]).", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\n\nIn the paper, the main motivation of the use of RL is to enable optimization of discrete objective. However, this use of RL is standard. Moreover, given this paper motivations, the reviewer would expect to see empirical comparisons to alternative methods to learn discrete assignments  (e.g. straight through, DETR-style https://arxiv.org/abs/2005.12872, [8]). Thus, from the methodological point of view the novelty is rather limited. From the application point of view, the instance segmentation of light microscopy images, the paper might have some edge of originality; however, it is hard to assess it given the niche domain. To fully appreciate the originality of the method it would be beneficial to validate the approach on a more standard benchmark. \n\n\n**Clarity:**\n\nThe paper is generally clearly written. However, the clarity of the methods could be improved. The presentation of this section is a bit convoluted making it hard to grasp the overall picture of the proposed approach. The reviewer would encourage the authors to simplify the presentation of the Methods, especially focusing on the aspects of why particular decisions were made. E.g. why are the authors considering method 1 and 2? Why stateless RL with sequence length of 1? Why not use straight through?\n\nFor a non-microscopy expert, it is hard to understand the color codings of Figs 4 and 5. \n\n\n**Quality and significance:**\n\nThe quality and significance of the results are hard to assess given the niche application domain. It would be beneficial to test the ideas on some more standard benchmarks to instance image segmentation (e.g. COCO). \n\nAdditional questions comments:\n- Why are superpixels a good prior for this task? \n- Would it be possible to use other priors/partitioning methods?\n- It would be nice to discuss the tradeoffs between added computational costs due to the use of RL and metric gains.\n- Would it be possible to report the numbers of a standard instance segmentation model for this task, (e.g. https://arxiv.org/abs/1703.06870 or some followup)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no limitation section/paragraph nor societal impact section/paragraph. Please add those to the conclusions section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}, {"review_id": "VMrg8k0AzVT", "reviewer": "Reviewer_dc1u", "summary": "In this paper, the authors proposed a reinforcement learning (RL) framework for instance segmentation with superpixels. The proposed framework is based on a stateless actor-critic setup and is demonstrated to be effective on the instance segmentation tasks for one toy dataset and one microscopy dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+: strength\n-: weakness \n\n1)Originality: \n\n+: This paper proposes to solve the instance segmentation tasks with weak supervision (superpixels) using the RL based methods. Different from the typical weakly supervised image instance segmentation methods, the RL-based one is a new and interesting application.\n\n-: The authors only talked about the related work on RL however ignore the related work on instance segmentation (fully supervised and weakly supervised). \n\n2) Quality:\n\n+: The proposed modules are proved to be effective via the ablation studies on the microscopy dataset. In addition, the authors also provide sufficient analysis on the selection of the hyper-parameters, such as the number of feature channels, random seed, etc.\n\n-: There lack extensive experiments on public instance segmentation benchmark datasets, such as MS COCO [a]. In addition, there are also many instance segmentation tasks in microscopy image analysis, such as nuclei instance segmentation in histopathology images [b], cell instance segmentation in electron microscopy images [1].\n\n[a] https://cocodataset.org/#home\n[b] A Multi-Organ Nucleus Segmentation Challenge, in IEEE TMI\n[1]: the reference #1 in the manuscript.\n\n3) Clarity:\n\n+: The methods and experiment sections are clearly presented.\n\n-: This paper aims at solving the weakly supervised instance segmentation via RL architecture, however, there lack illustrations and discussion on any related instance segmentation works.\n\n4) Significance:\n\n-: Although the authors have presented many experimental results, there still lack experimental results on representative benchmark instance segmentation datasets on general and medical images, which makes me feel that the overall paper lacks significance. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1) The major concern of this work is the lack of experimental results on public benchmark datasets. For the first toy dataset, the background content seems to be fixed (black content and white ‘noise’). For the second microscopy dataset, the morphological structure of the instances is always special: all the instances overlapping together to form a circle-like fruitfly embryo. To this end, the overall experiments on the two datasets are not sufficient to demonstrate the proposed method can be effective on other general datasets, such as the MS COCO. This made the overall impact of the proposed method questionable.  \n\n2) It seems that there are only 10 testing images in the real microscopy dataset, which is a small number. Therefore, a statistical significance test (based on p-value) is suggested for the results on this dataset.\n\n3) Computational complexity analysis is also suggested on the proposed framework.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "C-veejXhHU", "reviewer": "Reviewer_mhie", "summary": "This paper presents a strategy for object instance segmentation from images using training data that has no manual annotations. Instead of manual annotations, as is required for many state-of-the-art methods, this paper proposes to use high-level priors represented as rules that can be evaluated in the produced segmentation. These rules define what an acceptable segmentation is, and are used to create a reward signal for a reinforcement learning agent. The framework formulates an environment, an agent, an action space and a reward system in a stateless system to guide segmentation. All the components of the system are well designed and well thought, and this approach may have wide applications in bioimage segmentation problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The stateless actor-critic model has connections with the dynamics of Generative Adversarial Networks, as the authors note. The fact that there is no state also begs the question of what are the connections with the REINFORCE rule for training models when the loss function is non-differentiable, and how that approach could simplify or enhance the proposed formulation.\n* The experimental design is interesting and overall complete. However, the performance metrics used for evaluation are not in line with research in the instance segmentation field, where precision and recall are used in several different ways. For example, computing precision or F1 score at specific intersection over union thresholds would be more informative to understand how many objects / regions are correctly segmented and to what level of overlap with ground truth.\n* There are a few datasets studied in the instance segmentation literature that could have been used as benchmarks in this evaluation. At least one dataset used for nucleus segmentation of cells, for instance, would add a lot of value to the paper for two reasons: first, it connects with previous literature in the topic, second, it allows to assess how far a model trained without ground truth data would go for segmenting these objects. It is not expected that the proposed model should be better than fully supervised approaches, but being the first in the unsupervised regime would help understand how far we are from a competitive model while not requiring ground truth data.\n* The paper is overall well written and clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comments about limitations. Positive societal impacts expected, as less annotations would be required from experts to obtain accurate models.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use stateless RL for instance segmentation on microscopy datasets. Initially it received mixed reviews, upon the rebuttal and further discussion, reviewers are more aligned that the approach is novel, however they agreed that the benchmark dataset that the authors used were probably too small. The authors provided an update but that was after the discussion deadline so I don't think that can be considered for this round. Hence AC recommends rejection but the authors should consider adding experiments on public benchmark datasets and submit to the next available venue.
Decision: Reject